Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
Victor Stinner wrote: > I don't understand your email. Can you please elaborate? There is nothing wrong with the package. The remark is a joke provoked by a long history of a campaign [1] against external packages on distutils-sig. Many tools (like crate.io, when it was still up) have made derogatory remarks about external packages. Now the latest version of the officially sanctioned download tool (pip) spits out copious warnings, one of which is the subject of this thread. External packages are being singled out unfairly: 1) Anyone can upload any package to PyPI (i.e. the index is not curated at all). 2) Last time I looked, access credentials (via "lost login") were sent out in plaintext. 3) AFAIK people can upload a different (malicious) version of a package with *the exact same name*. 4) pip generally downloads the latest version, so a malicious person can provide a good package for several years until people trust him, then change to a trojaned version. 5) Looking at the list of certificates that is in my default cert store, I don't find SSL trustworthy at all. 6) D.J. Bernstein, who is somewhat security minded, has been shipping his software *for years* with just plain HTTP and published checksums. To sum it up: 1) Don't use pip to install packages directly from PyPI if security really matters. 2) The best security we currently get is either a) with package signatures (*if* you can get the author's key via a trustworthy channel, which is rarely the case). b) with decent checksums that are recorded on public mailing lists at the time the package is announced (it would be hard for an attacker to modify all mailing list archives.) Whether a package is internal or external is orthogonal to both points. With all these points, I find it questionable for an "official" install tool to make security related remarks about just one category of weaknesses. After all, people might be led to believe that pip is some sort of apt-get and all uploaded packages are safe. Stefan Krah [1] Note that the joke is quite innocent in comparison to what I've read on distutils-sig about the subject. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On May 8, 2014, at 8:12 AM, Stefan Krah wrote: > Victor Stinner wrote: >> I don't understand your email. Can you please elaborate? > > There is nothing wrong with the package. The remark is a joke provoked by > a long history of a campaign [1] against external packages on distutils-sig. > > Many tools (like crate.io, when it was still up) have made derogatory > remarks about external packages. Now the latest version of the officially > sanctioned download tool (pip) spits out copious warnings, one of which > is the subject of this thread. pip doesn't install externally hosted packaged by default because people should be aware of what servers they depend on. Installs that depend on externally hosted packages are brittle and more prone to failure. Every single external server adds *another* SPOF into any particular install set. Even if every external server has a 99.9% uptime, when you combine multiple of them the total uptime of any particular set of requirements drops quickly. This has been a major complaint that people have had over time. However they are not a security issue so they are easy to turn back on globally if a person wishes to make their own deployment more brittle (--allow-all-external). However in addition to externally hosted packages, there are also unverifiable packages. These are packages where there is no path to verify the download. Generally an unverifiable package is one that either is linked to directly but has no hash information encoded in the URL in a way that pip understands or, the more common case, it is a package that is linked from a page that is linked too on PyPI. Since we cannot build a path of trust to verify the package it is trivial for an attacker to MITM it and execute arbitrary Python code on the end users machine. This case *is* a security issue and pip goes out of it's way to *greatly* discourage depending on packages that require this kind of install. > > > External packages are being singled out unfairly: > > 1) Anyone can upload any package to PyPI (i.e. the index is not curated > at all). Sure, the threat model of PyPI isn't that you can install anything willy nilly and be safe. It's that when you ask for package X version Y, you should get what the *author* published and not what attacker who happens to be in a position to MITM you sends you. > > 2) Last time I looked, access credentials (via "lost login") were sent > out in plaintext. The existence of other security issues is not an excuse to not fix a security issue. There are other problems and we're slowly working on trying to clear them out. > > 3) AFAIK people can upload a different (malicious) version of a > package with *the exact same name*. Yes, a malicious author is needfully outside of the threat model for PyPI/pip. > > 4) pip generally downloads the latest version, so a malicious person > can provide a good package for several years until people trust > him, then change to a trojaned version. Yes, a malicious author is needfully outside of the threat model for PyPI/pip. > > 5) Looking at the list of certificates that is in my default cert > store, I don't find SSL trustworthy at all. TLS may not be the best method of authenticating package downloads, however it is an easy one and it is significantly better than having no authentication at all. You move your attack surface from "anyone who has a privileged network position" to "anyone who has a privileged network position and also has the ability to generate erroneously signed CA certificates". That is a significantly smaller attack surface. > > 6) D.J. Bernstein, who is somewhat security minded, has been shipping > his software *for years* with just plain HTTP and published checksums. Argument from authority doesn't really hold up very well when DJB doesn't distribute is software in a way that is intended to be consumed mechanically. Also while he may be a crypto expert he is far from an expert on successfully distributing software, unless you also think that the signature checking in most OS provided package managers is pointless. > > > To sum it up: > > 1) Don't use pip to install packages directly from PyPI if security > really matters. “security” isn’t a useful term on it’s own. You have to define a threat model. A better answer here is don’t use pip to install packages directly from PyPI if a malicious author is within your threat model (modulo other issue we’re still working on, but as I said above, the existence of a second security issue isn’t a reason not to fix the first issue). > > 2) The best security we currently get is either > > a) with package signatures (*if* you can get the author's key via > a trustworthy channel, which is rarely the case). > > b) with decent checksums that are recorded on public mailing > lists at the time the package is announced (it would be > hard for an attacker to modify all mailing list arch
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
Well, to be fair and leaving aside uptime concerns and the general desire to always install packages from some server instead of a safe and trusted local directory (probably too obvious ;-), it would certainly be possible to add support for trusted externally hosted packages. However, for some reason there's a strong resistance against doing this, which I frankly don't understand. I agree with Stefan that the warning message wording is less than ideal. You'd normally call such blanket statements FUD, esp. since there are plenty external hosting services which are reliable and safe to use. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 08 2014) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On Thu, May 8, 2014 at 11:39 PM, M.-A. Lemburg wrote: > I agree with Stefan that the warning message wording is less > than ideal. You'd normally call such blanket statements FUD, > esp. since there are plenty external hosting services which > are reliable and safe to use. No, it's not FUD. Every external dependency adds another thing that can fail. I was just arguing this with one of my younger brothers this evening; he had some stuff stored in Google Docs, and he couldn't access it because of some outage. With a local git/hg repository, he would have had all his data right there, and even losing access to a central hub server just means you can't pull/push. Using someone else's server makes you depend on that server and everything in between. Obviously that's often a cost worth paying (hey, I'm posting this from Gmail, so I completely lose access to all these emails if it's inaccessible), but it's a legitimate concern, not FUD. (That doesn't stop the wording from being "less than ideal", though.) ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On 8 May 2014 23:39, M.-A. Lemburg wrote: > However, for some reason there's a strong resistance against > doing this, which I frankly don't understand. Because we're taking responsibility for the end-to-end user experience of PyPI, and are expressly trying to eliminate the elements of that user experience that are beyond the control of the PyPI admin team (even the question of "does this software actually work?" is in our sights if you consider a long enough time span). That's hard enough with just a couple of service providers (Fastly and Rackspace) in the mix - it quickly becomes impossible if every new dependency from an installation request adds a new point of failure. Regards, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On May 8, 2014, at 9:39 AM, M.-A. Lemburg wrote: > Well, to be fair and leaving aside uptime concerns and the general > desire to always install packages from some server instead of > a safe and trusted local directory (probably too obvious ;-), > it would certainly be possible to add support for > trusted externally hosted packages. There is support for trusted externally hosted packages, you put the URL in PyPI and include a hash in the fragment like so: http://www.bytereef.org/software/mpdecimal/releases/cdecimal-2.3.tar.gz#md5=655f9fd72f7a21688f903900ebea6f56 The hash can be md5 or any of the sha-2 family. Now this does not mean that ``pip install cdecimal`` will automatically install this, because whether or not you're willing to install from servers other than PyPI[1] is a policy decision for the end user of pip. The only real contention point there is whether installing from other servers should be on or off by default. PEP438 selected off by default, and I agree with that decision. Installing externally hosted files, which are able to be safely downloaded[2], was a surprising behavior to *everyone* I've talked to who hadn't already discovered that pip/easy_install did that. For the people it wasn't surprising too, they said it was surprising when they had originally discovered it[3]. [1] To be specific, other than the configured index(es), which happens to default to PyPI. [2] For the definition of safe that PyPI/pip operate under, which is that the author of a package is assumed to be trusted by the person electing to download their package. [3] I suspect people who were around when PyPI *couldn't* host files and were only an index would be the exception to this. > > However, for some reason there's a strong resistance against > doing this, which I frankly don't understand. > > I agree with Stefan that the warning message wording is less > than ideal. You'd normally call such blanket statements FUD, > esp. since there are plenty external hosting services which > are reliable and safe to use. > I don't think the warning is FUD, and it doesn't mention anything security related at all. The exact text of the warning is in the subject of the email here: cdecimal an externally hosted file and may be unreliable Which is true as far as I can tell, it is externally hosted, and it may be unreliable[1]. If there is a better wording for that I’m happy to have it and will gladly commit it myself to pip. [1] In my experience dealing with complaints of pip's users, one of their big ones was that some dependency they use was, typically unknown to them, hosted externally and they found out it was hosted externally because the server it was hosted on went down. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On May 8, 2014, at 9:58 AM, Donald Stufft wrote: > Now this does not mean that ``pip install cdecimal`` will automatically > install > this, because whether or not you're willing to install from servers other than > PyPI[1] is a policy decision for the end user of pip. I forgot to add, for externally hosted files that are able to be safely downloaded pip's users can either elect to use any/all of them via: $ pip install --allow-all-external cdecimal $ PIP_ALLOW_ALL_EXTERNAL=1 pip install cdecimal $ echo "--allow-all-external\ncdecimal" > requirements.txt $ pip install -r requirements.txt $ echo "[global]\nallow-allow-external=true" > ~/.pip/pip.conf $ pip install cdecimal They can also elect to allow externally hosted files for *only* cdecimal using: $ pip install --allow-external cdecimal decimal $ PIP_ALLOW_EXTERNAL=cdecimal pip install cdecimal $ echo "--allow-external decimal\ncdecimal" > requirements.txt $ pip install -r requirements.txt $ echo "[global]\nallow-external=decimal" > ~/.pip/pip.conf $ pip install cdecimal I may have the syntax on the non command flag options slightly wrong, those are auto generated for us in our option system and I don't personally use those options. Also the reason for the extra verbosity in ``--allow-external`` is so you can allow it on something you don't directly depend on, like: $ pip install --allow-external cdecimal foobar-which-depends-on-cdecimal - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On Thu, 08 May 2014 09:58:08 -0400, Donald Stufft wrote: > I don't think the warning is FUD, and it doesn't mention anything security > related at all. The exact text of the warning is in the subject of the email > here: > > cdecimal an externally hosted file and may be unreliable > > Which is true as far as I can tell, it is externally hosted, and it may be > unreliable[1]. If there is a better wording for that Iâm happy to have it > and > will gladly commit it myself to pip. > > [1] In my experience dealing with complaints of pip's users, one of their big > ones was that some dependency they use was, typically unknown to them, > hosted externally and they found out it was hosted externally because the > server it was hosted on went down. "unreliable" reads as "not safe", ie: insecure. You probably want something like "and access to it may be unreliable". --David ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On May 8, 2014, at 10:11 AM, R. David Murray wrote: > On Thu, 08 May 2014 09:58:08 -0400, Donald Stufft wrote: >> I don't think the warning is FUD, and it doesn't mention anything security >> related at all. The exact text of the warning is in the subject of the email >> here: >> >>cdecimal an externally hosted file and may be unreliable >> >> Which is true as far as I can tell, it is externally hosted, and it may be >> unreliable[1]. If there is a better wording for that I’m happy to have it and >> will gladly commit it myself to pip. >> >> [1] In my experience dealing with complaints of pip's users, one of their big >>ones was that some dependency they use was, typically unknown to them, >>hosted externally and they found out it was hosted externally because the >>server it was hosted on went down. > > "unreliable" reads as "not safe", ie: insecure. > > You probably want something like "and access to it may be unreliable". > > --David Done: https://github.com/pypa/pip/commit/69bf7067 - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On Thu, 08 May 2014 10:11:39 -0400, "R. David Murray" wrote: > On Thu, 08 May 2014 09:58:08 -0400, Donald Stufft wrote: > > I don't think the warning is FUD, and it doesn't mention anything security > > related at all. The exact text of the warning is in the subject of the email > > here: > > > > cdecimal an externally hosted file and may be unreliable > > > > Which is true as far as I can tell, it is externally hosted, and it may be > > unreliable[1]. If there is a better wording for that IâÂÂm happy to have > > it and > > will gladly commit it myself to pip. > > > > [1] In my experience dealing with complaints of pip's users, one of their > > big > > ones was that some dependency they use was, typically unknown to them, > > hosted externally and they found out it was hosted externally because > > the > > server it was hosted on went down. > > "unreliable" reads as "not safe", ie: insecure. > > You probably want something like "and access to it may be unreliable". Actually, thinking about this some more, *most* end-users aren't going to care that there's another point of failure here, they only care if it works or not when they try to install it. So something like "cdecimal is not hosted on pypi; download may fail if external server is unavailable" might be clearer. And once you're at that point, as a user I'm going to grumble, "Well, why the heck didn't you just try?", as I figure out how to re-execute the command so that it does try. --David ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] EuroPython CPython Sprint?
Hi all, What do you think about a CPython sprint at EuroPython 2014? Regards, Stephane -- Stéphane Wirtel - http://wirtel.be - @matrixise ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On Thu, 08 May 2014 10:21:34 -0400 "R. David Murray" wrote: > > > > "unreliable" reads as "not safe", ie: insecure. > > > > You probably want something like "and access to it may be unreliable". > > Actually, thinking about this some more, *most* end-users aren't going > to care that there's another point of failure here, they only care if it > works or not when they try to install it. So something like > "cdecimal is not hosted on pypi; download may fail if external server > is unavailable" might be clearer. > > And once you're at that point, as a user I'm going to grumble, "Well, why > the heck didn't you just try?", as I figure out how to re-execute the > command so that it does try. Agreed. That warning looks rather pointless and only aimed at trying to enforce the pip developers' ideological preferences. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] EuroPython CPython Sprint?
On Thu May 08 2014 at 10:25:44 AM, Stéphane Wirtel wrote: > Hi all, > > What do you think about a CPython sprint at EuroPython 2014? > Great, although I think that answer would be considered obvious since there is no real negative to holding sprints. =) Are you indirectly asking if anyone plans to lead such a sprint? -Brett > > Regards, > > Stephane > -- > Stéphane Wirtel - http://wirtel.be - @matrixise > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > brett%40python.org > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On May 8, 2014, at 10:21 AM, R. David Murray wrote: > On Thu, 08 May 2014 10:11:39 -0400, "R. David Murray" > wrote: >> On Thu, 08 May 2014 09:58:08 -0400, Donald Stufft wrote: >>> I don't think the warning is FUD, and it doesn't mention anything security >>> related at all. The exact text of the warning is in the subject of the email >>> here: >>> >>>cdecimal an externally hosted file and may be unreliable >>> >>> Which is true as far as I can tell, it is externally hosted, and it may be >>> unreliable[1]. If there is a better wording for that Iâm happy to have it >>> and >>> will gladly commit it myself to pip. >>> >>> [1] In my experience dealing with complaints of pip's users, one of their >>> big >>>ones was that some dependency they use was, typically unknown to them, >>>hosted externally and they found out it was hosted externally because the >>>server it was hosted on went down. >> >> "unreliable" reads as "not safe", ie: insecure. >> >> You probably want something like "and access to it may be unreliable". > > Actually, thinking about this some more, *most* end-users aren't going > to care that there's another point of failure here, they only care if it > works or not when they try to install it. So something like > "cdecimal is not hosted on pypi; download may fail if external server > is unavailable" might be clearer. > > And once you're at that point, as a user I'm going to grumble, "Well, why > the heck didn't you just try?", as I figure out how to re-execute the > command so that it does try. > > --David > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/donald%40stufft.io Most users are not going to care up until the point where the external server is unavailable, and then they care a whole lot. On the tin it sounds reasonable to just download the external file if the server is up however we've done that for a long time and the end result has been end user pain. Now requiring someone to add a flag in order to download an externally hosted file is also end user pain. The difference between the two pains is when they happen. The requiring a flag pain happens at the point of decision, when you decide to make your deployment depend on something hosted externally. The default to allow pain happens sometime in the future, when you may or may not have any idea why suddenly your installs aren't working (and when you look, PyPI is up so you're really very confused why this particular file doesn't work). Even worse is the case when a project has some old files, but the newer versions aren't hosted and suddenly people are getting very old releases which is even more confusing to the end users. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
Donald Stufft wrote: > There is support for trusted externally hosted packages, you put the URL in > PyPI and include a hash in the fragment like so: > > http://www.bytereef.org/software/mpdecimal/releases/cdecimal-2.3.tar.gz#md5=655f9fd72f7a21688f903900ebea6f56 That is exactly the mode I was using until today. This mode produced the subject's warning message. Today I've switched to manual install mode with manual sha256sum verification which is *far* safer than anything you get via pip right now. > [2] For the definition of safe that PyPI/pip operate under, which is that the > author of a package is assumed to be trusted by the person electing to > download their package. No, there are other holes, which you have conceded in your previous mail. > I don't think the warning is FUD, and it doesn't mention anything security > related at all. The exact text of the warning is in the subject of the email > here: > > cdecimal an externally hosted file and may be unreliable > > Which is true as far as I can tell, it is externally hosted, and it may be > unreliable[1]. If there is a better wording for that I?m happy to have it and > will gladly commit it myself to pip. Do you honestly not see a difference between the cited warning and the *intended* warning "the server's availability may be unreliable"? Even the latter is FUD or a truism (it applies to any server). The real question is: Why is there a warning if the person running pip has explicitly allowed external packages? Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On May 8, 2014, at 10:31 AM, Antoine Pitrou wrote: > On Thu, 08 May 2014 10:21:34 -0400 > "R. David Murray" wrote: >>> >>> "unreliable" reads as "not safe", ie: insecure. >>> >>> You probably want something like "and access to it may be unreliable". >> >> Actually, thinking about this some more, *most* end-users aren't going >> to care that there's another point of failure here, they only care if it >> works or not when they try to install it. So something like >> "cdecimal is not hosted on pypi; download may fail if external server >> is unavailable" might be clearer. >> >> And once you're at that point, as a user I'm going to grumble, "Well, why >> the heck didn't you just try?", as I figure out how to re-execute the >> command so that it does try. > > Agreed. That warning looks rather pointless and only aimed at trying to > enforce the pip developers' ideological preferences. > > Regards > > Antoine. > The pip developers didn’t make this decision. It was discussed on distutils-sig hammered out in a PEP, and then accepted. We took part in that discussion, but ultimately we implemented PEP438. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On 08.05.2014 15:58, Donald Stufft wrote: > > On May 8, 2014, at 9:39 AM, M.-A. Lemburg wrote: > >> Well, to be fair and leaving aside uptime concerns and the general >> desire to always install packages from some server instead of >> a safe and trusted local directory (probably too obvious ;-), >> it would certainly be possible to add support for >> trusted externally hosted packages. > > There is support for trusted externally hosted packages, you put the URL in > PyPI and include a hash in the fragment like so: > > http://www.bytereef.org/software/mpdecimal/releases/cdecimal-2.3.tar.gz#md5=655f9fd72f7a21688f903900ebea6f56 > > The hash can be md5 or any of the sha-2 family. > > Now this does not mean that ``pip install cdecimal`` will automatically > install > this, because whether or not you're willing to install from servers other than > PyPI[1] is a policy decision for the end user of pip. Hmm, if you call that feature "trusted externally hosted packages", pip should really do trust them, right ? ;-) I can understand that pip defaults to not trusting URLs which don't meet the above feature requirements, but not that it still warns about unreliable externally hosted packages even if the above feature is used. At the moment, pip will refuse to use an externally hosted files even if the package author uses the above hashed URLs; even with HTTPS and proper SSL certificate chain. > The only real contention > point there is whether installing from other servers should be on or off by > default. PEP438 selected off by default, and I agree with that decision. > Installing externally hosted files, which are able to be safely downloaded[2], > was a surprising behavior to *everyone* I've talked to who hadn't already > discovered that pip/easy_install did that. For the people it wasn't surprising > too, they said it was surprising when they had originally discovered it[3]. > > [1] To be specific, other than the configured index(es), which happens to > default to PyPI. > [2] For the definition of safe that PyPI/pip operate under, which is that the > author of a package is assumed to be trusted by the person electing to > download their package. > [3] I suspect people who were around when PyPI *couldn't* host files and were > only an index would be the exception to this. > >> >> However, for some reason there's a strong resistance against >> doing this, which I frankly don't understand. >> >> I agree with Stefan that the warning message wording is less >> than ideal. You'd normally call such blanket statements FUD, >> esp. since there are plenty external hosting services which >> are reliable and safe to use. >> > > I don't think the warning is FUD, and it doesn't mention anything security > related at all. The exact text of the warning is in the subject of the email > here: > > cdecimal an externally hosted file and may be unreliable > > Which is true as far as I can tell, it is externally hosted, and it may be > unreliable[1]. If there is a better wording for that I’m happy to have it and > will gladly commit it myself to pip. The current version of pip writes: Downloading/unpacking pkg Could not find any downloads that satisfy the requirement pkg Some externally hosted files were ignored (use --allow-external pkg to allow). Cleaning up... No distributions at all found for pkg This wording if fine, IMO. The wording Stefan quoted gets generated for dependencies. This should probably be changed to the same wording (including the reference to the right command line option to use). > [1] In my experience dealing with complaints of pip's users, one of their big > ones was that some dependency they use was, typically unknown to them, > hosted externally and they found out it was hosted externally because the > server it was hosted on went down. I think that's a general problem, not one of some server being down: users put too much trust into the dependencies of packages they use. Regardless of how safe/reliable we make things w/r to file hosting, this problem does not go away. It's just too easy for people to get tricked into trusting packages they don't even know about. Nothing we'll ever change, though. People are lazy and easily drop all such concerns for ease of use :-( -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 08 2014) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On 08.05.2014 15:57, Nick Coghlan wrote: > On 8 May 2014 23:39, M.-A. Lemburg wrote: >> However, for some reason there's a strong resistance against >> doing this, which I frankly don't understand. > > Because we're taking responsibility for the end-to-end user experience > of PyPI, and are expressly trying to eliminate the elements of that > user experience that are beyond the control of the PyPI admin team Oh, I guess you'd have to rewrite most of those 40k packages then :-) Seriously, the word "eliminate" in there does not sit well with our goals for openness. External services like github, sourceforge, bitbucket, dropbox, cdns, etc. are not per-se evil and unreliable. pip should acknowledge this and not try to "eliminate" all hosting services in the world per default [sound of Empire Strikes Back theme] ;-) > (even the question of "does this software actually work?" is in our > sights if you consider a long enough time span). That's hard enough > with just a couple of service providers (Fastly and Rackspace) in the > mix - it quickly becomes impossible if every new dependency from an > installation request adds a new point of failure. Like I said: the best option is to use a local directory which only contains packages files that you have inspected and actually trust :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 08 2014) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On May 8, 2014, at 10:36 AM, Stefan Krah wrote: > Donald Stufft wrote: >> There is support for trusted externally hosted packages, you put the URL in >> PyPI and include a hash in the fragment like so: >> >> http://www.bytereef.org/software/mpdecimal/releases/cdecimal-2.3.tar.gz#md5=655f9fd72f7a21688f903900ebea6f56 > > That is exactly the mode I was using until today. This mode produced the > subject's warning message. > > Today I've switched to manual install mode with manual sha256sum verification > which is *far* safer than anything you get via pip right now. It is not safer in any meaingful way. If someone is in a position to compromise the integrity of PyPI's TLS, they can replace the hash on that page with something else. Now you've attempted to work around this by telling people to go look up the release announcement hash. However if someone can compromise the integrity of PyPI's TLS, they can also compromise the integrity of https://mail.python.org/, or GMane, or any other TLS based website[1]. All of that assumes that the end user is going to bother to verify the hash *at all* which almost none of them will and they'll just check the http url into their requirements.txt file and be downloading things over HTTP and be vulnerable to arbitrary code execution via MITM. > > >> [2] For the definition of safe that PyPI/pip operate under, which is that the >>author of a package is assumed to be trusted by the person electing to >>download their package. > > No, there are other holes, which you have conceded in your previous mail. The presence of other holes is not a useful argument to avoid closing a hole. We're working to close all of them, and that ends up meaning we close one at a time. > > >> I don't think the warning is FUD, and it doesn't mention anything security >> related at all. The exact text of the warning is in the subject of the email >> here: >> >>cdecimal an externally hosted file and may be unreliable >> >> Which is true as far as I can tell, it is externally hosted, and it may be >> unreliable[1]. If there is a better wording for that I?m happy to have it and >> will gladly commit it myself to pip. > > Do you honestly not see a difference between the cited warning and the > *intended* warning "the server's availability may be unreliable”? Do I? No I don’t. However I’ve since adjusted the message based on R David Murray’s feedback to make sure it specifically says that access may be unreliable instead of just that the package itself may be unreliable. > > Even the latter is FUD or a truism (it applies to any server). No, because the use of an external host *adds* additional unreliability. If PyPI is down, then all packages are down, including ones that host externally. If the cdecimal server is down, then that one specific package is unavailable. It is impossible to do anything but reduce the overall availability by adding additional SPOFs. > > The real question is: Why is there a warning if the person running pip > has explicitly allowed external packages? > Why is there a warning? Originally that warning was there because the first part of the transition to this "mode" of defaults was to give an option to disable externally hosted files, but leave it on by default. In this phase we gave this warning to tell the people who just leave things to their default about this file. Should the warning itself still exist? I don't know, but a better avenue for asking for a change in pip is via our issue tracker instead of whining on python-dev. > Stefan Krah > > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/donald%40stufft.io - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] EuroPython CPython Sprint?
On 8 May 2014, at 16:33, Brett Cannon wrote: > On Thu May 08 2014 at 10:25:44 AM, Stéphane Wirtel > wrote: > >> Hi all, >> >> What do you think about a CPython sprint at EuroPython 2014? >> > > Great, although I think that answer would be considered obvious since there > is no real negative to holding sprints. =) Are you indirectly asking if > anyone plans to lead such a sprint? Yes, I do, I can make the request to the EuroPython team for the sprint but: * I am not a Python Core Dev * I can't find the right issues for the sprinters * I can't help the sprinters with the issues. * I am a newbie in the CPython code Thus, who would be interested in a CPython sprint? > > -Brett Stéphane, -- Stéphane Wirtel - http://wirtel.be - @matrixise ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On 9 May 2014 00:52, "M.-A. Lemburg" wrote: > > On 08.05.2014 15:57, Nick Coghlan wrote: > > > (even the question of "does this software actually work?" is in our > > sights if you consider a long enough time span). That's hard enough > > with just a couple of service providers (Fastly and Rackspace) in the > > mix - it quickly becomes impossible if every new dependency from an > > installation request adds a new point of failure. > > Like I said: the best option is to use a local directory which > only contains packages files that you have inspected and > actually trust :-) Correct, but that raises the barrier to entry too high. The pip defaults are aimed at providing an experience with the fewest points of failure that is currently achievable, with a minimal learning curve. We still have a long way to go, but if people want to influence those design decisions, the relevant lists are pypa-dev (for pip specific discussions) and distutils-sig (for higher level cross-tool design decisions) Cheers, Nick. > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Source (#1, May 08 2014) > >>> Python Projects, Consulting and Support ... http://www.egenix.com/ > >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ > >>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ > > > : Try our mxODBC.Connect Python Database Interface for free ! :: > >eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg >Registered at Amtsgericht Duesseldorf: HRB 46611 >http://www.egenix.com/company/contact/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
Donald Stufft wrote: > hosted packages are brittle and more prone to failure. Every single external > server adds *another* SPOF into any particular install set. Even if every > external server has a 99.9% uptime, when you combine multiple of them the > total > uptime of any particular set of requirements drops quickly. This has been a > major complaint that people have had over time. We have been through that many times; to me this is an indication that people are using pip under circumstances when they should not. pip is not apt-get. [I am aware that *you* know that, just stating it again for the broader audience.] > > 2) Last time I looked, access credentials (via "lost login") were sent > > out in plaintext. > > The existence of other security issues is not an excuse to not fix a security > issue. There are other problems and we're slowly working on trying to clear > them out. It is, however, a reason to avoid error messages that could *imply* (rightly or wrongly) to users that the combination of pip and internal packages is safe. > > 3) AFAIK people can upload a different (malicious) version of a > > package with *the exact same name*. > > Yes, a malicious author is needfully outside of the threat model for PyPI/pip. How so? I'm avoiding this attack by publishing sha256sums at release time. The point is that I *cannot* change cdecimal-2.3.tar.gz without a user digging up a checksum mismatch from the mailing list archives. > > 6) D.J. Bernstein, who is somewhat security minded, has been shipping > > his software *for years* with just plain HTTP and published checksums. > > Argument from authority doesn't really hold up very well when DJB doesn't > distribute is software in a way that is intended to be consumed mechanically. > Also while he may be a crypto expert he is far from an expert on successfully > distributing software, unless you also think that the signature checking in > most OS provided package managers is pointless. That is sort of a strawman. The whole point *is* that certain distribution models don't lend themselves to mechanical consumption. I cannot speak for DJB, perhaps he is just thinking that GPG signing is pointless if users can't validate the signature and SSL is pointless because one cannot trust governments. OS package signing is useful since the packages are curated. If anyone could upload a package to Debian, whereupon it would be signed with the official key, apt-get would lose its usefulness. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On Thu, 08 May 2014 10:37:15 -0400, Donald Stufft wrote: > Most users are not going to care up until the point where the external server > is unavailable, and then they care a whole lot. On the tin it sounds > reasonable > to just download the external file if the server is up however we've done > that for a long time and the end result has been end user pain. > > Now requiring someone to add a flag in order to download an externally hosted > file is also end user pain. The difference between the two pains is when they > happen. The requiring a flag pain happens at the point of decision, when you > decide to make your deployment depend on something hosted externally. The > default to allow pain happens sometime in the future, when you may or may not > have any idea why suddenly your installs aren't working (and when you look, > PyPI is up so you're really very confused why this particular file doesn't > work). Even worse is the case when a project has some old files, but the newer > versions aren't hosted and suddenly people are getting very old releases which > is even more confusing to the end users. Ah, I understand now. Your perspective is as someone who is using pip for *deployment*. I'm speaking from a python+plus+pip end-user perspective, which is going to be even more common now that it is part of the Python distribution. I'm not sure how you reconcile these two worlds. I would venture to suggest that if you are using it for deployment you really ought to be using a local package repository[*], not the global one; but, as someone observed, the sensible thing to do and what people actually do are often very different; and, since I haven't done this particular kind of deployment scenario in Python myself, there may be reasons I'm missing. However, your last mention of "end users" confuses me. Why are "end users" getting old packages in a deployment situation? Isn't it the developer/operations people (and the latter would, I assume, control the 'external packages' flag) who would be seeing that? Maybe you mean something by deployment different from how I use the word? --David [*] I found it *such* a pain to do this for perl/cpan. I have a project for a customer where I have to do this, and the hoops I had to jump through to get a reliable deployment (where packages wouldn't be unexpectedly upgraded under my feet) were nasty. (This was several years ago and I haven't revisited it, so maybe things have gotten better, or I just missed something.) I haven't had to do it for python yet, oddly enough, so I don't know how hard it is for Python. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On May 8, 2014, at 11:19 AM, Stefan Krah wrote: > Donald Stufft wrote: >> hosted packages are brittle and more prone to failure. Every single external >> server adds *another* SPOF into any particular install set. Even if every >> external server has a 99.9% uptime, when you combine multiple of them the >> total >> uptime of any particular set of requirements drops quickly. This has been a >> major complaint that people have had over time. > > We have been through that many times; to me this is an indication that > people are using pip under circumstances when they should not. pip is > not apt-get. > > [I am aware that *you* know that, just stating it again for the broader > audience.] > > >>> 2) Last time I looked, access credentials (via "lost login") were sent >>>out in plaintext. >> >> The existence of other security issues is not an excuse to not fix a security >> issue. There are other problems and we're slowly working on trying to clear >> them out. > > It is, however, a reason to avoid error messages that could *imply* (rightly > or wrongly) to users that the combination of pip and internal packages is > safe. > > >>> 3) AFAIK people can upload a different (malicious) version of a >>>package with *the exact same name*. >> >> Yes, a malicious author is needfully outside of the threat model for >> PyPI/pip. > > How so? I'm avoiding this attack by publishing sha256sums at release time. > The point is that I *cannot* change cdecimal-2.3.tar.gz without a user digging > up a checksum mismatch from the mailing list archives. Practically speaking exactly zero people will ever bother to dig up the checksum from a mailing list archive, or even verify the hash that you have right on PyPI itself. Worse security that people actually use is infinitely better than better security that nobody uses. > > >>> 6) D.J. Bernstein, who is somewhat security minded, has been shipping >>>his software *for years* with just plain HTTP and published checksums. >> >> Argument from authority doesn't really hold up very well when DJB doesn't >> distribute is software in a way that is intended to be consumed mechanically. >> Also while he may be a crypto expert he is far from an expert on successfully >> distributing software, unless you also think that the signature checking in >> most OS provided package managers is pointless. > > That is sort of a strawman. The whole point *is* that certain distribution > models don't lend themselves to mechanical consumption. I cannot speak > for DJB, perhaps he is just thinking that GPG signing is pointless if > users can't validate the signature and SSL is pointless because one cannot > trust governments. > > OS package signing is useful since the packages are curated. If anyone > could upload a package to Debian, whereupon it would be signed with the > official key, apt-get would lose its usefulness. People are going to mechanically consume them. You can tell them it’s wrong all you want but they are going to do it. > > > > Stefan Krah > > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/donald%40stufft.io - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On May 8, 2014, at 11:21 AM, R. David Murray wrote: > On Thu, 08 May 2014 10:37:15 -0400, Donald Stufft wrote: >> Most users are not going to care up until the point where the external server >> is unavailable, and then they care a whole lot. On the tin it sounds >> reasonable >> to just download the external file if the server is up however we've done >> that for a long time and the end result has been end user pain. >> >> Now requiring someone to add a flag in order to download an externally hosted >> file is also end user pain. The difference between the two pains is when they >> happen. The requiring a flag pain happens at the point of decision, when you >> decide to make your deployment depend on something hosted externally. The >> default to allow pain happens sometime in the future, when you may or may not >> have any idea why suddenly your installs aren't working (and when you look, >> PyPI is up so you're really very confused why this particular file doesn't >> work). Even worse is the case when a project has some old files, but the >> newer >> versions aren't hosted and suddenly people are getting very old releases >> which >> is even more confusing to the end users. > > Ah, I understand now. > > Your perspective is as someone who is using pip for *deployment*. Deployment, or any kind of situation where you want to have a reproducible build. Generally via deployment yes. > > I'm speaking from a python+plus+pip end-user perspective, which is going > to be even more common now that it is part of the Python distribution. > > I'm not sure how you reconcile these two worlds. I would venture to > suggest that if you are using it for deployment you really ought to > be using a local package repository[*], not the global one; but, as > someone observed, the sensible thing to do and what people actually > do are often very different; and, since I haven't done this particular > kind of deployment scenario in Python myself, there may be reasons > I'm missing. People simply don’t do this[1]. Especially in a world with things like Heroku existing which makes it stupid simple to use pip to install from PyPI but installing from your own server requires standing up some infrastructure. > > However, your last mention of "end users" confuses me. Why are "end > users" getting old packages in a deployment situation? Isn't it the > developer/operations people (and the latter would, I assume, control > the 'external packages' flag) who would be seeing that? Maybe you mean > something by deployment different from how I use the word? Someone using pip, this may be a developer who is just trying to recreate their production environment locally, this may be someone using chef/puppet to automate installing via pip, this may be someone pushing to Heroku. The old versions thing is more that it’s really confusing when you type ``pip install foo`` on a monday and get 2.0, and ``pip install foo`` on weds and get 0.4. > > --David > > [*] I found it *such* a pain to do this for perl/cpan. I have a > project for a customer where I have to do this, and the hoops I had > to jump through to get a reliable deployment (where packages wouldn't > be unexpectedly upgraded under my feet) were nasty. (This was several > years ago and I haven't revisited it, so maybe things have gotten better, > or I just missed something.) > > I haven't had to do it for python yet, oddly enough, so I don't know > how hard it is for Python. For Python with pip you can use a requirements.txt file to create a set of dependencies that are pinned to exact versions like: foo==2.0 bar==2.3 And pip will (theoretically, our dep solving is real bad ATM) install exactly those versions from your index server. Generally this means PyPI which means the author can delete the version and upload a new file with the same version number. However it’s also trivial to stand up your own server. It can be as easy as pointing nginx/Apache at a static directory with autoindex = True. (See: https://wheels.caremad.io/). On top of that there is peep which adds a secure message digest on it to make sure that the author/index didn’t swap things out on you, and there is some discussion about how best to add that to pip itself. > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/donald%40stufft.io - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
Donald Stufft wrote: > > Today I've switched to manual install mode with manual sha256sum > > verification > > which is *far* safer than anything you get via pip right now. > > It is not safer in any meaingful way. > > If someone is in a position to compromise the integrity of PyPI's TLS, they > can replace the hash on that page with something else. Now you've attempted to > work around this by telling people to go look up the release announcement > hash. However if someone can compromise the integrity of PyPI's TLS, they can > also compromise the integrity of https://mail.python.org/, or GMane, or any > other TLS based website[1]. Of course it is safer. Suppose a file is stored on PyPI: 1) Attacker guesses my username (or is it even visible, I'm not sure). 2) Clicks on "lost login". 3) Intercepts mail (difficult, but far from the TLS attack category). Maybe on a home or university network. Or a rogue person at a mail provider. 4) Changes the uploaded file together with the hash. pip would be perfectly happy, checking the hash via Google would turn up a mismatch. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On 08.05.2014 16:42, M.-A. Lemburg wrote: > On 08.05.2014 15:58, Donald Stufft wrote: >> >> On May 8, 2014, at 9:39 AM, M.-A. Lemburg wrote: >> >>> Well, to be fair and leaving aside uptime concerns and the general >>> desire to always install packages from some server instead of >>> a safe and trusted local directory (probably too obvious ;-), >>> it would certainly be possible to add support for >>> trusted externally hosted packages. >> >> There is support for trusted externally hosted packages, you put the URL in >> PyPI and include a hash in the fragment like so: >> >> http://www.bytereef.org/software/mpdecimal/releases/cdecimal-2.3.tar.gz#md5=655f9fd72f7a21688f903900ebea6f56 >> >> The hash can be md5 or any of the sha-2 family. >> >> Now this does not mean that ``pip install cdecimal`` will automatically >> install >> this, because whether or not you're willing to install from servers other >> than >> PyPI[1] is a policy decision for the end user of pip. > > Hmm, if you call that feature "trusted externally hosted packages", > pip should really do trust them, right ? ;-) > > I can understand that pip defaults to not trusting URLs which don't > meet the above feature requirements, but not that it still warns > about unreliable externally hosted packages even if the above > feature is used. > > At the moment, pip will refuse to use an externally hosted files even > if the package author uses the above hashed URLs; even with HTTPS > and proper SSL certificate chain. Could this perhaps be changed/reconsidered for pip ? Note that easy_install/setuptools does not have such problems. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 08 2014) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On May 8, 2014, at 11:34 AM, Stefan Krah wrote: > Donald Stufft wrote: >>> Today I've switched to manual install mode with manual sha256sum >>> verification >>> which is *far* safer than anything you get via pip right now. >> >> It is not safer in any meaingful way. >> >> If someone is in a position to compromise the integrity of PyPI's TLS, they >> can replace the hash on that page with something else. Now you've attempted >> to >> work around this by telling people to go look up the release announcement >> hash. However if someone can compromise the integrity of PyPI's TLS, they can >> also compromise the integrity of https://mail.python.org/, or GMane, or any >> other TLS based website[1]. > > Of course it is safer. Suppose a file is stored on PyPI: > > 1) Attacker guesses my username (or is it even visible, I'm not sure). > > 2) Clicks on "lost login". > > 3) Intercepts mail (difficult, but far from the TLS attack category). > Maybe on a home or university network. Or a rogue person at a > mail provider. > > 4) Changes the uploaded file together with the hash. > > > pip would be perfectly happy, checking the hash via Google would turn > up a mismatch. I said “meaningful”. Almost nobody is going to ever bother googling it and the likelihood that someone is able to MITM *you* specifically is far lesser than the likelihood that someone is going to MITM one of the cdecimal users. Additionally your messages aren’t signed and email isn’t an authenticated profile so if someone was able to get your password they could simply spoof and email from you to the mailing list with new hashes, or edit out the description telling people to go google some stuff. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On May 8, 2014, at 11:37 AM, M.-A. Lemburg wrote: > On 08.05.2014 16:42, M.-A. Lemburg wrote: >> On 08.05.2014 15:58, Donald Stufft wrote: >>> >>> On May 8, 2014, at 9:39 AM, M.-A. Lemburg wrote: >>> Well, to be fair and leaving aside uptime concerns and the general desire to always install packages from some server instead of a safe and trusted local directory (probably too obvious ;-), it would certainly be possible to add support for trusted externally hosted packages. >>> >>> There is support for trusted externally hosted packages, you put the URL in >>> PyPI and include a hash in the fragment like so: >>> >>> http://www.bytereef.org/software/mpdecimal/releases/cdecimal-2.3.tar.gz#md5=655f9fd72f7a21688f903900ebea6f56 >>> >>> The hash can be md5 or any of the sha-2 family. >>> >>> Now this does not mean that ``pip install cdecimal`` will automatically >>> install >>> this, because whether or not you're willing to install from servers other >>> than >>> PyPI[1] is a policy decision for the end user of pip. >> >> Hmm, if you call that feature "trusted externally hosted packages", >> pip should really do trust them, right ? ;-) >> >> I can understand that pip defaults to not trusting URLs which don't >> meet the above feature requirements, but not that it still warns >> about unreliable externally hosted packages even if the above >> feature is used. >> >> At the moment, pip will refuse to use an externally hosted files even >> if the package author uses the above hashed URLs; even with HTTPS >> and proper SSL certificate chain. > > Could this perhaps be changed/reconsidered for pip ? > > Note that easy_install/setuptools does not have such problems. Anything can be changes or reconsidered of course. I feel pretty strongly that an installer should not install things from places other than the index without a specific opt in. That discussion would be best done on distutils-sig as it would require reversing the decision in PEP438. I really don't feel strongly one way or the other about the *warning* that happens when you allow an external file. It exists primarily because at the time it was implemented external files were default to allowed. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
Donald Stufft wrote: > I said ?meaningful?. Almost nobody is going to ever bother googling it and > the likelihood that someone is able to MITM *you* specifically is far lesser > than the likelihood that someone is going to MITM one of the cdecimal users. I'm doing this for important installs. -- That is how I installed qmail and djbdns. > Additionally your messages aren?t signed and email isn?t an authenticated > profile so if someone was able to get your password they could simply spoof > and email from you to the mailing list with new hashes, or edit out the > description > telling people to go google some stuff. Signing messages is pointless if the key isn't well connected. Also, I'm reading the lists and would notice a "release". Most importantly, the checksum mismatch would still be found, since the old messages with the correct sum would still exist under the scenario we're talking about (i.e. not GHCQ hacking into Belgacom routers). Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On May 8, 2014, at 12:03 PM, Stefan Krah wrote: > Donald Stufft wrote: >> I said ?meaningful?. Almost nobody is going to ever bother googling it and >> the likelihood that someone is able to MITM *you* specifically is far lesser >> than the likelihood that someone is going to MITM one of the cdecimal users. > > I'm doing this for important installs. -- That is how I installed qmail > and djbdns. > > >> Additionally your messages aren?t signed and email isn?t an authenticated >> profile so if someone was able to get your password they could simply spoof >> and email from you to the mailing list with new hashes, or edit out the >> description >> telling people to go google some stuff. > > Signing messages is pointless if the key isn't well connected. Also, I'm > reading the lists and would notice a "release". Most importantly, the > checksum mismatch would still be found, since the old messages with the > correct sum would still exist under the scenario we're talking about > (i.e. not GHCQ hacking into Belgacom routers). > > I’m unsure if you’re being willfully dense or if you’re just not understanding what I mean when I say “almost”. Of course there are going to be a few outliers where people do bother to do that, but it’s not going to be common place at all. But whatever, I’ve removed the warning that occurs when you install an externally hosted file [1] and it will be included in pip 1.6. I have not changed the defaults for --allow-all-external nor have I removed the warning that occurs when someone elects to install an unverifiable download. [1] https://github.com/pypa/pip/commit/9f56b79e8d - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On Thu, 08 May 2014 11:32:28 -0400, Donald Stufft wrote: > On May 8, 2014, at 11:21 AM, R. David Murray wrote: > > Ah, I understand now. > > > > Your perspective is as someone who is using pip for *deployment*. > > Deployment, or any kind of situation where you want to have a reproducible > build. Generally via deployment yes. [...] > For Python with pip you can use a requirements.txt file to create a set of > dependencies that are pinned to exact versions like: > > foo==2.0 > bar==2.3 > > And pip will (theoretically, our dep solving is real bad ATM) install exactly > those versions from your index server. Generally this means PyPI which OK, this makes sense, then. (I wish perl/cpan had something similar...maybe it does, but I couldn't find it at the time.) This still leaves the fact that there is a disconnect between the "needs" of two different audiences for PIP: people who deploy things, and everyone else who just uses pip to install stuff. The second group is going to overwhelm the first group, if it doesn't already. And I think that's all the comments I have on this issue :) --David ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On May 8, 2014, at 12:42 PM, R. David Murray wrote: > On Thu, 08 May 2014 11:32:28 -0400, Donald Stufft wrote: >> On May 8, 2014, at 11:21 AM, R. David Murray wrote: >>> Ah, I understand now. >>> >>> Your perspective is as someone who is using pip for *deployment*. >> >> Deployment, or any kind of situation where you want to have a reproducible >> build. Generally via deployment yes. > [...] >> For Python with pip you can use a requirements.txt file to create a set of >> dependencies that are pinned to exact versions like: >> >> foo==2.0 >> bar==2.3 >> >> And pip will (theoretically, our dep solving is real bad ATM) install exactly >> those versions from your index server. Generally this means PyPI which > > OK, this makes sense, then. (I wish perl/cpan had something > similar...maybe it does, but I couldn't find it at the time.) > > This still leaves the fact that there is a disconnect between the > "needs" of two different audiences for PIP: people who deploy things, > and everyone else who just uses pip to install stuff. Yup balancing between the two is something we have to do in every decision we make. When PEP438 was being discussed I did a pretty extensive amount of investigation into what affect this change would have [1]. What I found was that: - The sizable majority was projects would host things on PyPI - There was a significant chunk of projects where a single release or two would only be available externally and it was an accident that they weren’t uploaded. - Of the links that were available externally, very few of them were available in a way that was verifiable and were thus insecure to install. Because of this it was determined that simply allowing externally hosted files without also allowing externally hosted and unverified files would not actually have a significant impact for the vast bulk of the projects that were not hosted on PyPI. > > The second group is going to overwhelm the first group, if it doesn't > already. Generally yes, because not every who uses pip to deploy uses pip to install locally, but most people who use pip to deploy also use pip locally. > > And I think that's all the comments I have on this issue :) [1]: https://github.com/dstufft/pypi.linkcheck - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Tix version needed to build 2.7 Windows installer?
This is mostly a question for Martin, but perhaps someone else would also know. I'm trying to build the 2.7 installers so I can backport the path option from 3.3, but I can't seem to figure out which version of Tix is necessary to have a complete build. So far any of them on http://svn.python.org/projects/external don't appear to be configured to work with the combination of tcl and tk that are used on 2.7, per the buildbot external scripts. It's another issue that Tix isn't even downloaded by the scripts, but I'll do it manually right now just to get this going. Any tips? ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tix version needed to build 2.7 Windows installer?
Am 08.05.14 18:59, schrieb Brian Curtin: > This is mostly a question for Martin, but perhaps someone else would also > know. > > I'm trying to build the 2.7 installers so I can backport the path > option from 3.3, but I can't seem to figure out which version of Tix > is necessary to have a complete build. So far any of them on > http://svn.python.org/projects/external don't appear to be configured > to work with the combination of tcl and tk that are used on 2.7, per > the buildbot external scripts. It's another issue that Tix isn't even > downloaded by the scripts, but I'll do it manually right now just to > get this going. > > Any tips? Ah, 2.7. I was using a checkout of tix-8.4.3.x from Nov 13, 2010, one where python.mak was pointing to Tcl 8.5.2. It may not have been tagged. In 2.7, it wasn't checked out out automatically because Tix requires the original Tcl/Tk path names, so building it automatically would have failed. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On 8 May 2014 16:46, Donald Stufft wrote: > Anything can be changes or reconsidered of course. I feel pretty strongly that > an installer should not install things from places other than the index > without > a specific opt in. That discussion would be best done on distutils-sig as it > would require reversing the decision in PEP438. I think it's worth reconsidering. Since this behaviour was implemented, there have been many instances of confusion and unhappiness with the situation, both from package developers and pip users. I don't think that's good for pip. I would like to see PEP 438 reviewed with the intention of working out how to fix the user experience (ideally while retaining the reliability enhancements, but accepting that compromises may be needed). Some points: 1. Often a user won't know that a package is externally hosted until an install fails. Having to add a flag and retry is a lousy experience. Why not at a minimum have an "externally hosted - are you sure?" prompt? 2. The way the flags are implemented (notably the need to repeat the package name) is clearly confusing and irritating for users, even if the reasons are logical. We should look at fixing that. 3. The user complaints against pip are significant and ongoing. I don't think they should be ignored. If PEP 438 cannot be reconsidered in the light of user feedback, then I think the pip developers may need to have a discussion about whether we conform to the PEP (clearly Donald thinks we should, I'm not sure I do, and I don't know about the others). But I don't think it's healthy for pip to be looking at deliberately ignoring an accepted PEP, so I'd prefer it if the debate was around addressing the issues in the PEP itself. 4. Maybe distutils-sig *isn't* the right place to discuss revising PEP 438. The issues getting raised are end-user problems, and the users most affected are unlikely to be on that list. Socially, this change does not seem to be having the effect of persuading more package developers to host on PyPI. The stick doesn't appear to have worked, maybe we should be trying to find a carrot? Or maybe we have to accept that some developers have sound reasons for not hosting on PyPI and work with them to find an acceptable compromise? Has anyone checked what Stefan's reasons are for not hosting cdecimal on PyPI? Do they represent a use case that the PEP hasn't considered? > I really don't feel strongly one way or the other about the *warning* that > happens when you allow an external file. It exists primarily because at the > time it was implemented external files were default to allowed. I think it's reasonable to remove the warning. If the user chooses to allow an external file, it makes sense to assume they understand the implications and not nag them about their decision. Particularly given the level of controversy the warning is generating. On a personal note, I'm uncomfortable with the way this change is perceived as a case of *pip* enforcing a behaviour that the pip developers feel should be required. I actually don't like this change particularly. So having pip implement the behaviour required by that PEP is to me simply a case of compliance with the agreed standard. But now, as a pip developer, being held responsible for the resulting user pain, and being expected to defend it, does not make me happy. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On May 8, 2014, at 5:02 PM, Paul Moore wrote: > On 8 May 2014 16:46, Donald Stufft wrote: >> Anything can be changes or reconsidered of course. I feel pretty strongly >> that >> an installer should not install things from places other than the index >> without >> a specific opt in. That discussion would be best done on distutils-sig as it >> would require reversing the decision in PEP438. > > I think it's worth reconsidering. Since this behaviour was > implemented, there have been many instances of confusion and > unhappiness with the situation, both from package developers and pip > users. I don't think that's good for pip. I would like to see PEP 438 > reviewed with the intention of working out how to fix the user > experience (ideally while retaining the reliability enhancements, but > accepting that compromises may be needed). I think most of the confusion has been over the fact that —allow-external takes a package name, not that it exists at all. > > Some points: > > 1. Often a user won't know that a package is externally hosted until > an install fails. Having to add a flag and retry is a lousy > experience. Why not at a minimum have an "externally hosted - are you > sure?" prompt? A prompt is OK with me. > 2. The way the flags are implemented (notably the need to repeat the > package name) is clearly confusing and irritating for users, even if > the reasons are logical. We should look at fixing that. Yea, there’s a ticket for this. > 3. The user complaints against pip are significant and ongoing. I > don't think they should be ignored. If PEP 438 cannot be reconsidered > in the light of user feedback, then I think the pip developers may > need to have a discussion about whether we conform to the PEP (clearly > Donald thinks we should, I'm not sure I do, and I don't know about the > others). But I don't think it's healthy for pip to be looking at > deliberately ignoring an accepted PEP, so I'd prefer it if the debate > was around addressing the issues in the PEP itself. I don’t think the problem with with the PEP. > 4. Maybe distutils-sig *isn't* the right place to discuss revising PEP > 438. The issues getting raised are end-user problems, and the users > most affected are unlikely to be on that list. > > Socially, this change does not seem to be having the effect of > persuading more package developers to host on PyPI. The stick doesn't > appear to have worked, maybe we should be trying to find a carrot? Do you have any data to point to that says it hasn’t worked? Just to see what impact it has had, I’m running my scripts again that I ran a year ago to see what has changed, already I can see they are processing MUCH faster than last year. > Or > maybe we have to accept that some developers have sound reasons for > not hosting on PyPI and work with them to find an acceptable > compromise? Has anyone checked what Stefan's reasons are for not > hosting cdecimal on PyPI? Do they represent a use case that the PEP > hasn't considered? If I recall correctly his reasoning is that he finds the legal requirements associated with uploading to PyPI to be unsatisfactory. > >> I really don't feel strongly one way or the other about the *warning* that >> happens when you allow an external file. It exists primarily because at the >> time it was implemented external files were default to allowed. > > I think it's reasonable to remove the warning. If the user chooses to > allow an external file, it makes sense to assume they understand the > implications and not nag them about their decision. Particularly given > the level of controversy the warning is generating. The warning is gone as of a few hours ago. > > On a personal note, I'm uncomfortable with the way this change is > perceived as a case of *pip* enforcing a behaviour that the pip > developers feel should be required. I actually don't like this change > particularly. So having pip implement the behaviour required by that > PEP is to me simply a case of compliance with the agreed standard. But > now, as a pip developer, being held responsible for the resulting user > pain, and being expected to defend it, does not make me happy. > > Paul I think the pain is being overrepresented and the positives are being ignored. The problem is the benefits of this PEP are much like the benefits of TLS too. For the vast majority of people they don’t notice anything different except installing things is faster and more reliable. They don’t attribute that to the PEP or this decision, they just internalize it as the new norm. However the people who this does affect will seek out why it broke and raise an issue citing that thing specifically. This creates a perception of lots of pain for no gain when the reality is not that. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail _
Re: [Python-Dev] Tix version needed to build 2.7 Windows installer?
On Thu, May 8, 2014 at 2:36 PM, "Martin v. Löwis" wrote: > Am 08.05.14 18:59, schrieb Brian Curtin: >> This is mostly a question for Martin, but perhaps someone else would also >> know. >> >> I'm trying to build the 2.7 installers so I can backport the path >> option from 3.3, but I can't seem to figure out which version of Tix >> is necessary to have a complete build. So far any of them on >> http://svn.python.org/projects/external don't appear to be configured >> to work with the combination of tcl and tk that are used on 2.7, per >> the buildbot external scripts. It's another issue that Tix isn't even >> downloaded by the scripts, but I'll do it manually right now just to >> get this going. >> >> Any tips? > > Ah, 2.7. I was using a checkout of tix-8.4.3.x from Nov 13, 2010, one > where python.mak was pointing to Tcl 8.5.2. It may not have been tagged. > > In 2.7, it wasn't checked out out automatically because Tix requires the > original Tcl/Tk path names, so building it automatically would have failed. I updated the 2.7 buildbot scripts to pull in Tcl/Tk 8.5.15 a couple of weeks ago (see http://bugs.python.org/issue21303), but hadn't gotten anything done with Tix yet. It should just need python.mak updated to point to Tcl/Tk 8.5.15; it's on my list to get fixed as soon as I can. -- Zach ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On 9 May 2014 07:23, "Donald Stufft" wrote: > On May 8, 2014, at 5:02 PM, Paul Moore wrote: > > > Or > > maybe we have to accept that some developers have sound reasons for > > not hosting on PyPI and work with them to find an acceptable > > compromise? Has anyone checked what Stefan's reasons are for not > > hosting cdecimal on PyPI? Do they represent a use case that the PEP > > hasn't considered? > > If I recall correctly his reasoning is that he finds the legal requirements > associated with uploading to PyPI to be unsatisfactory. I actually need to follow up on that, because the terms *were* legally questionable last time I looked (also too hard to review, since as far as I am aware, they're only presented during new user sign-up). I'll deal with that at work today. Cheers, Nick. P.S. Still the wrong list, folks. Most of the people you need to convince to get changes made to pip, PyPI and the packaging ecosystem in general aren't here. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On May 8, 2014, at 6:20 PM, Nick Coghlan wrote: > > On 9 May 2014 07:23, "Donald Stufft" wrote: > > On May 8, 2014, at 5:02 PM, Paul Moore wrote: > > > > > Or > > > maybe we have to accept that some developers have sound reasons for > > > not hosting on PyPI and work with them to find an acceptable > > > compromise? Has anyone checked what Stefan's reasons are for not > > > hosting cdecimal on PyPI? Do they represent a use case that the PEP > > > hasn't considered? > > > > If I recall correctly his reasoning is that he finds the legal requirements > > associated with uploading to PyPI to be unsatisfactory. > > I actually need to follow up on that, because the terms *were* legally > questionable last time I looked (also too hard to review, since as far as I > am aware, they're only presented during new user sign-up). > > I'll deal with that at work today. > > > I’m pretty sure VanL wrote the terms and has explicitly said they won’t change and are exactly as broad as they need to be without being any broader[1]. They are linked to from the footer of every UI centric PyPI page. [1] https://mail.python.org/pipermail/python-legal-sig/2013-March/03.html - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On 9 May 2014 08:22, "Donald Stufft" wrote: > > > On May 8, 2014, at 6:20 PM, Nick Coghlan wrote: >> >> I actually need to follow up on that, because the terms *were* legally questionable last time I looked (also too hard to review, since as far as I am aware, they're only presented during new user sign-up). >> >> I'll deal with that at work today. >> >> > I’m pretty sure VanL wrote the terms and has explicitly said they won’t change and are exactly as broad as they need to be without being any broader[1]. They are linked to from the footer of every UI centric PyPI page. Thanks for the additional references. It should be possible to clarify the terms to address Red Hat's (and other users') concerns while still addressing the points Van is worried about (for example, adding a footnote explaining the use cases that need to be covered, and being explicit that users must respect *both* licenses). However, this subthread is now even further off-topic for this list, so I'll take it to python-legal-sig later today with some specific proposed wording changes. Cheers, Nick. > > [1] https://mail.python.org/pipermail/python-legal-sig/2013-March/03.html > > - > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On May 8, 2014, at 5:22 PM, Donald Stufft wrote: >> Socially, this change does not seem to be having the effect of >> persuading more package developers to host on PyPI. The stick doesn't >> appear to have worked, maybe we should be trying to find a carrot? > > Do you have any data to point to that says it hasn’t worked? Just to see > what impact it has had, I’m running my scripts again that I ran a year > ago to see what has changed, already I can see they are processing > MUCH faster than last year. The data has finished processing, it represents a time diff of approximately one year. The pip release that caused all of this was released about 4-5 months ago. Overall PyPI has seen a 50% growth in installable projects in that time. If the change would have had no effect we'd expect to see a ~50% increase across the board. However what we've seen is a a 60% (+10% of expected) increase in projects that can only be installed from PyPI and a 12% decrease in projects that have any unsafe files (-62% of expected). Further more we can see that if pip were to change the default of --allow-all-external it would take 23 projects from unable to be installed by default to able to be installed by default. This represents 0.2% of installable projects on PyPI. It would take an additional 40 projects and make one or more additional files able to be downloaded by default. Some other data points: * We've gone from 86% of projects being installable from PyPI to 92%. * We've gone from 5% of projects being only unsafely installable to 3% * We've gone from 14% of projects having any files unsafe to install to 8% * We've gone from 0.004% of projects being safely hosted externally to 0.2% Looking at these numbers I think it's safe to say that in this time period that the "hosting hygiene" of a PyPI project is more likely to be a better state than it was a year ago. We cannot state for a fact if this is because of this change or not, however given that the fallout is ~23 (or ~63) projects out of 38,835 I think it is incredibly reasonable to leave the defaults alone since there is a reasonably high chance that they played at least some part in that change. I'd love to get these numbers to the point where the number of projects installable strictly from PyPI is 100% (or at least 100% installable safely), however 92% (or 92.2%) is getting pretty close to that and hopefully that number will just continue to grow until it hits 100%. For reference, here's the raw numbers as well as some summary of the data here: https://gist.github.com/dstufft/b14008d11c0a5760dbed And the repository where the raw data as well as the scripts used to collect and process it is here: https://github.com/dstufft/pypi.linkcheck linkcollector.py collections while linkwriter.py writes out the json file, and stats2.py processes and gives the numbers from the gist above. links.json is the data from a year ago, and 2014-05-08.links.json is the data from today. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On May 9, 2014, at 12:34 AM, Donald Stufft wrote: > The data has finished processing, it represents a time diff of approximately > one year. The pip release that caused all of this was released about 4-5 > months > ago. Oh I forgot to mention: In order to make the comparison as accurate as possible I've used the same collection script as I did a year ago. This is prior to the real advent of Wheels so these numbers do not take into accounts Wheels at all (which pip can also install) but it *does* include Eggs which pip cannot install. Further more it also includes #egg=dev urls which point to a VCS (and it would consider those an unverifiable/unsafe download). - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
