Re: [Distutils] a plea for backward-compatibility / smooth transitions
On Tue, Jul 30, 2013 at 4:58 PM, Donald Stufft don...@stufft.io wrote: Hrm. So I hear what you're saying and part of the problem is likely due to the history of where I tried to get a change through and then felt like all I was getting was stop energy and people wanting to keep the status quo which ultimately ended up preventing changes has lead me to view distutils-sig in more of an adversarial light then is probably appropriate for the distutils-sig of 2013 (versus the distutils-sig of 2011/2012). This is probably reflected in my tone and likely has others, such as yourself, respond similarly, pushing us further down that path. My thought process has become Ok here's something that needs to happen, now how do I get distutils-sig not to prevent it from happening. Thanks for the thoughtful response. I appreciate it. I also want to just throw in one extra piece of information for you and anybody else reading this: 99% of stop energy doesn't happen because people actively want to prevent progress or frustrate other people. It simply happens when people notice a problem but don't have as much personal stake in your goal as they do in not experiencing the problem they will experience (or perceive they will), from the proposed change. When you look at it from this perspective, it's easier to understand that the way to fix this is with more engagement on their part, which can only be gotten by engagement on your part. When I first proposed WSGI, the initial reaction of Web-SIG was pretty negative. Stop energy if you will. Things only moved forward once I was able to channel the energy of objections into offering solutions. It's helpful to remember that asking, okay, so how you would recommend I do it? *doesn't* obligate you to actually follow all of the recommendations you get. (Especially since some of them will be mutually contradictory!) Anyway, I guess what I'm saying is that people lacking enthusiasm for your goals is not really them trying to stop you. In fact, objections are a positive thing: it means you got somebody's attention. The next step is to leverage that attention into actually getting help, or at least more constructive input. ;-) It's true that some individuals will never provide really helpful input. In the WSGI effort, there were people whose advice I never took, because their goals were directed entirely opposite to where I wanted to go. But I remained engaged until it was mutually clear (or at least I thought it was) that our goals were different, and didn't try to persuade them to go in the same direction. Such attempts at persuasion are pretty much a waste of time, and a big energy drain. Consensus-building is something that you do with people who have at least some goals in common, so it's best to focus on finding out what you *do* agree on. This was again reflected in the Python 2.3 discussion because my immediate reaction and impression was that you were attempting to block the move from MD5 due to Python 2.3, which I felt 2.3 wasn't worth blocking enhancements to PyPI. The snark in my statements primarily came from that feeling of someone was trying to shut down an enhancement. Right. In such a case, a question you could ask is, Do you agree in general that we should move to a better hash at some point in the future?, because then the disagreement can be narrowed down to timeframe, migration or deprecation process, etc. The truth is, I had no intention of blocking the move, I had concerns I wanted addressed about the impact, timing and process. (Actually, I originally just noticed a couple of errors in what you'd laid out as the current state of things, and wanted to make sure they were included in the assessment.) The point is, if somebody doesn't have *any* common ground with you, it's unlikely they're even talking to you. At the very least, if they're talking with you about PyPI, they must care about PyPI, even if they care about different things than you, or with different relative priorities. ;-) As far as how to fix it I don't have a particularly magic answer. I will try to be more mindful of my tone and that distutils-sig is likely not my adversary anymore as well as try to ask questions instead of arguing the relevance immediately. Again, thank you. And hopefully, remember that probably nobody was intentionally being your adversary before, either. As the old adage says, the best way to destroy your enemies is to make friends with them. ;-) And we do that by focusing on common ground, and inviting participation. (This is again not to say that I've been 100% Mr. Wonderful myself; I know I haven't. But the community's best consensus-building happens when somebody is doing the tough work of engaging with all parties. Sometimes this doesn't happen, alas; back when I was developing setuptools there just weren't enough people interested in the problems available on Distutils-SIG to build any sort of consensus on
Re: [Distutils] a plea for backward-compatibility / smooth transitions
On Aug 2, 2013, at 4:28 PM, PJ Eby p...@telecommunity.com wrote: I also want to just throw in one extra piece of information for you and anybody else reading this: 99% of stop energy doesn't happen because people actively want to prevent progress or frustrate other people. It simply happens when people notice a problem but don't have as much personal stake in your goal as they do in not experiencing the problem they will experience (or perceive they will), from the proposed change. One of the biggest things I viewed as stop energy was how impossible it seemed to get a SSL certificate that was trusted in most major browsers to be used on PyPI. There was never any reason given that we couldn't move to one, just simply we already have a certificate without addressing the reasons I was trying to get a different one. Beyond that it was other random proposals that it felt like people put in the minimal amount of effort in order to find a reason to say no to it. Then if I addressed that particular concern, they'd again seem to put in the minimal amount of effort to find a reason to say no. It felt like the goal posts were just constantly being moved to slightly further away each time. That sort of behavior is mostly what I mean by stop energy. It's entirely frustrating and it feels like it's entire purpose is to dissuade someone from changing the status quo, wether that was the intention or not. Everything else you said has been noted :) - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] a plea for backward-compatibility / smooth transitions
On Jul 31, 2013, at 1:38 AM, holger krekel hol...@merlinux.eu wrote: Hi Donald, On Tue, Jul 30, 2013 at 14:04 -0400, Donald Stufft wrote: On Jul 30, 2013, at 1:13 PM, PJ Eby p...@telecommunity.com wrote: On Tue, Jul 30, 2013 at 4:14 AM, Donald Stufft don...@stufft.io wrote: Heh, I'm pretty good at getting yelled at :) Nick is also pretty good at making people feel like he both knows and *cares* about their breakage, and isn't just dismissing their concerns as trivial or unimportant. Breakage isn't trivial or unimportant to the person who's yelling, so this is an important community-maintenance skill. It builds trust, and reduces the total amount of yelling. *shrug*, If I didn't care I would have made this change as soon as Nick said it was ok. Instead I declared I was going to and waited to make sure nobody else had any concerns. And once Holger said he did I said ok I won't do it. Maybe my mannerisms give the impression I don't but that's actually pretty far from the truth. For this particular change I originally created the pip commit that allowed it, and then again I created the setuptools commit, backporting hashlib into setuptools to support Python 2.4. I put a decent amount of effort into trying to make sure that nothing broke but in the end there were still concerns :) For the record, i am all for putting generic hash support into the installers and maybe prepare for an eventual change to make PyPI serve sha256 hashes. However, to me it's not clear if such a move may become obsolete through the potential advent of TUF. pip has had generic hash support since 1.2, setuptools since 0.9, and since I believe zc.buildout just lets setuptools do the downloads so it'd be when using 0.9 for it. The idea was this was an easy win security hardening. Something like TUF would more or less obsolete it but TUF isn't on the immediate radar. My original objection reason was tied to generally pushing for more focus on backward-compatibility. I am grateful that several people including you, Nick and Jannis acknowledged the point. best, holger - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] a plea for backward-compatibility / smooth transitions
On Jul 29, 2013, at 10:41 PM, Antoine Pitrou solip...@pitrou.net wrote: Paul Moore p.f.moore at gmail.com writes: Personally, none of the changes have detrimentally affected me, so my opinion is largely theoretical. But even I am getting a little frustrated by the constant claims that what we have now is insecure and broken, and must be fixed ASAP. FWIW, +1. You may be paranoid, but not everyone has to be (or suffer the consequences of it). Security issues should be fixed without breaking things in a hassle (which is the policy we followed e.g. for the ssl module, or hash randomization). You missed a key word … when possible. If there is a problem we will fix it, when we can do that in a way that minimizes breakages we will do that. Its all just about cost-benefit, and when you are talking about executing code downloaded from the internet it becomes quite easy to see benefits outweighing costs even with pretty major UX changes. Not something we do lightly, but status quo does not win here, sorry. The whole python.org infrastructure is built on an OS kernel written by someone who thinks security issues are normal bugs. AFAIK there is no plan to switch to OpenBSD. This is news to me, we specifically run Ubuntu LTS because Canonical's security response team has a proven track record of handling issues. If you mean that Linus doesn't handle security issues well, then it is fortunate indeed that we don't actually use his software. --Noah signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] a plea for backward-compatibility / smooth transitions
On Jul 30, 2013, at 1:41 AM, Antoine Pitrou solip...@pitrou.net wrote: Paul Moore p.f.moore at gmail.com writes: Personally, none of the changes have detrimentally affected me, so my opinion is largely theoretical. But even I am getting a little frustrated by the constant claims that what we have now is insecure and broken, and must be fixed ASAP. FWIW, +1. You may be paranoid, but not everyone has to be (or suffer the consequences of it). Security issues should be fixed without breaking things in a hassle (which is the policy we followed e.g. for the ssl module, or hash randomization). People are generally not paranoid until they've been successfully attacked. I *will* advocate and push for breaking things where security is concerned because regardless of if you care or not, a lot of people *do* care and the nature of the beast is that you're only as strong as the weakest link. This particular change wasn't an immediate vulnerability that I felt was urgent, hence why I've backed off on it when people were concerned about the backwards compat implications. I will not back off when it comes to issues that *do* have an immediate or near term issue, regardless of if some people don't care or not. The whole python.org infrastructure is built on an OS kernel written by someone who thinks security issues are normal bugs. AFAIK there is no plan to switch to OpenBSD. So classifying bugs as security vs normal is supposed to make it easier on people. The thought is that creating new releases and applying updates is a time consuming process and often times requires things such as reboots or service restarts so by dividing issues up into security vs not security the amount of disruption can be minimized for only important updates. There's actually pretty strong evidence that shows the process of classifying bugs as security bugs is a harmful process and that all updates should be treated the same because it's often times not immediately obvious what the security implications are, even to security experts[1]. I'm sure your dig at the OS is supposed to be some sort of masterstroke about how we're not being as secure as possible anyways however I would contest that OpenBSD is actually more secure. It's major claim to fame is that they haven't had a vulnerably in the OpenBSD base system in a heck of a long time. The problem is the OpenBSD base system is terribly small and that claim cannot be made once you include their packages. Further more at the last I checked OpenBSD does not provide (although this may have changed) and abilities to do MAC which means you're relying entirely on an attackers ability to *not* get in versus providing fail safes to contain an attack once it's happened. Infrastructure is not using MAC currently but I would love to get us to that point as well. [1] citeseerx.ist.psu.edu/viewdoc/download;jsessionid=7B6E224144709E99B7FAEBFC497621A1?doi=10.1.1.148.9757rep=rep1type=pdf - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] a plea for backward-compatibility / smooth transitions
On Jul 30, 2013, at 2:02 AM, Noah Kantrowitz n...@coderanger.net wrote: On Jul 29, 2013, at 10:41 PM, Antoine Pitrou solip...@pitrou.net wrote: Paul Moore p.f.moore at gmail.com writes: Personally, none of the changes have detrimentally affected me, so my opinion is largely theoretical. But even I am getting a little frustrated by the constant claims that what we have now is insecure and broken, and must be fixed ASAP. FWIW, +1. You may be paranoid, but not everyone has to be (or suffer the consequences of it). Security issues should be fixed without breaking things in a hassle (which is the policy we followed e.g. for the ssl module, or hash randomization). You missed a key word … when possible. If there is a problem we will fix it, when we can do that in a way that minimizes breakages we will do that. Its all just about cost-benefit, and when you are talking about executing code downloaded from the internet it becomes quite easy to see benefits outweighing costs even with pretty major UX changes. Not something we do lightly, but status quo does not win here, sorry. Basically said it better than I could :) - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] a plea for backward-compatibility / smooth transitions
Noah Kantrowitz noah at coderanger.net writes: The whole python.org infrastructure is built on an OS kernel written by someone who thinks security issues are normal bugs. AFAIK there is no plan to switch to OpenBSD. This is news to me, we specifically run Ubuntu LTS because Canonical's security response team has a proven track record of handling issues. If you mean that Linus doesn't handle security issues well, then it is fortunate indeed that we don't actually use his software. Did you already forget what the discussion is about? Security/bugfix Ubuntu LTS updates don't break compatibility for the sake of hardening things, which is the whole point. (As for the idea that using Canonical's kernel amounts to not using Linus' software, that's a rather unorthodox notion of authorship. It's very likely Canonical doesn't change more than 1% LOC in the kernel, so you're still bound to Linus' decisions for at least 99% of the code - and even probably for the remaining 1%, since Canonical's version won't be massively divergent.) Regards Antoine. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] a plea for backward-compatibility / smooth transitions
On Jul 29, 2013, at 11:19 PM, Antoine Pitrou solip...@pitrou.net wrote: Noah Kantrowitz noah at coderanger.net writes: The whole python.org infrastructure is built on an OS kernel written by someone who thinks security issues are normal bugs. AFAIK there is no plan to switch to OpenBSD. This is news to me, we specifically run Ubuntu LTS because Canonical's security response team has a proven track record of handling issues. If you mean that Linus doesn't handle security issues well, then it is fortunate indeed that we don't actually use his software. Did you already forget what the discussion is about? Security/bugfix Ubuntu LTS updates don't break compatibility for the sake of hardening things, which is the whole point. Again, speaking as the guy that has to clean up the mess when they do break compat, I promise you they do. Same deal, they only break compat when keeping compat would present a threat to users, which is quite often the case with security bugs. They are fortunately a bit further ahead of us on the long tail of finding problems, so this is far less frequent than it was in years past. We will get there too, but like I said, status quo is not a defense here, just strap in and hang on. --Noah signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] a plea for backward-compatibility / smooth transitions
Donald Stufft donald at stufft.io writes: I *will* advocate and push for breaking things where security is concerned because regardless of if you care or not, a lot of people *do* care and the nature of the beast is that you're only as strong as the weakest link. That's nice, but you're not alone here, so whatever you want to push for needn't always happen. There's actually pretty strong evidence that shows the process of classifying bugs as security bugs is a harmful process and that all updates should be treated the same because it's often times not immediately obvious what the security implications are, even to security experts[1]. Doesn't it contradict your own stance on the subject? (This shows a fundamental misunderstanding of how security issues present themselves. Of course things just work for people because security issues are not like regular bugs - which is a flawed argument btw. Many bugs have random or rare occurrences - not just security issues) I'm sure your dig at the OS is supposed to be some sort of masterstroke about how we're not being as secure as possible anyways however I would contest that OpenBSD is actually more secure. WTF are you talking about? No it's not. I'm simply pointing out that, for some strange reason, you decided to trust an OS whose author has very different views on how to fix security issues than you have. Regards Antoine. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] a plea for backward-compatibility / smooth transitions
On Jul 30, 2013, at 2:19 AM, Antoine Pitrou solip...@pitrou.net wrote: Noah Kantrowitz noah at coderanger.net writes: The whole python.org infrastructure is built on an OS kernel written by someone who thinks security issues are normal bugs. AFAIK there is no plan to switch to OpenBSD. This is news to me, we specifically run Ubuntu LTS because Canonical's security response team has a proven track record of handling issues. If you mean that Linus doesn't handle security issues well, then it is fortunate indeed that we don't actually use his software. Did you already forget what the discussion is about? Security/bugfix Ubuntu LTS updates don't break compatibility for the sake of hardening things, which is the whole point. Well for one PyPI doesn't have releases so there is no LTS or not LTS, it's just what's being served so there's no simple place to break backwards compatibility. As far as forgetting what's being discussed here then it sounds like you've apparently missed the fact I already conceded the change to MD5 and further more this thread was explicitly split off from the MD5 request because, as far as I can tell, Holger wanted to discuss the broader topic of compatibility in general and not just specific to this particular issue. (As for the idea that using Canonical's kernel amounts to not using Linus' software, that's a rather unorthodox notion of authorship. It's very likely Canonical doesn't change more than 1% LOC in the kernel, so you're still bound to Linus' decisions for at least 99% of the code - and even probably for the remaining 1%, since Canonical's version won't be massively divergent.) Regards Antoine. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] a plea for backward-compatibility / smooth transitions
Donald Stufft donald at stufft.io writes: I have zero qualms about releasing a full disclosure along with working exploits into the wild for a security vulnerability that people block me on. If I'm unable to rectify the problem I will make sure that everyone *knows* about the problem. I don't know what I'm supposed to infer from such a statement, except that I probably don't want to trust you. You might think that publish[ing] working exploits into the wild is some kind of heroic, altruistic act, but I think few people would agree. Even a random occurrence will break for some percentage of people using the software some percentage of the time. If it didn't then it's unlikely anyone would notice it. Security vulnerabilities typically won't break until someone actively tries to break them. You're mistaken. Bugs can sometimes be fixed preemptively, even before they're noticed in the wild (by means of perusing the code and noticing an issue, for example). Which also includes, of course, security issues (which often get fixed before they ever get exploited). Regards Antoine. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] a plea for backward-compatibility / smooth transitions
On Jul 30, 2013, at 12:01 AM, Antoine Pitrou solip...@pitrou.net wrote: Donald Stufft donald at stufft.io writes: I have zero qualms about releasing a full disclosure along with working exploits into the wild for a security vulnerability that people block me on. If I'm unable to rectify the problem I will make sure that everyone *knows* about the problem. I don't know what I'm supposed to infer from such a statement, except that I probably don't want to trust you. You might think that publish[ing] working exploits into the wild is some kind of heroic, altruistic act, but I think few people would agree. No, this is the standard for security researchers. If the vendor ignores the reported exploit for long enough, they go public and try to make sure users understand the risks and how to mitigate them in the time it takes the vendor to fix it. --Noah signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] a plea for backward-compatibility / smooth transitions
On 30 July 2013 17:33, Paul Moore p.f.mo...@gmail.com wrote: And in case it's not obvious, I think this is important. We need to have this sort of debate, certainly, but it won't happen without someone advocating (and implementing!) the changes, so many thanks for being that person and putting up with the flak. Part of the idea of having myself and Richard in the approval chain for packaging ecosystem and PyPI changes is to encourage people to get angry at *us* for saying Yes, let's do that when things break, rather than getting mad at the people actually volunteering to do the work needed to improve the Python community's software distribution infrastructure. That's the plan, anyway, even if it isn't always all that effective in practice :( Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] a plea for backward-compatibility / smooth transitions
On Jul 30, 2013, at 4:10 AM, Nick Coghlan ncogh...@gmail.com wrote: On 30 July 2013 17:33, Paul Moore p.f.mo...@gmail.com wrote: And in case it's not obvious, I think this is important. We need to have this sort of debate, certainly, but it won't happen without someone advocating (and implementing!) the changes, so many thanks for being that person and putting up with the flak. Part of the idea of having myself and Richard in the approval chain for packaging ecosystem and PyPI changes is to encourage people to get angry at *us* for saying Yes, let's do that when things break, rather than getting mad at the people actually volunteering to do the work needed to improve the Python community's software distribution infrastructure. That's the plan, anyway, even if it isn't always all that effective in practice :( Heh, I'm pretty good at getting yelled at :) - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] a plea for backward-compatibility / smooth transitions
Donald Stufft donald at stufft.io writes: On Jul 30, 2013, at 3:01 AM, Antoine Pitrou solipsis at pitrou.net wrote: I don't know what I'm supposed to infer from such a statement, except that Iprobably don't want to trust you. You might think that publish[ing] workingexploits into the wild is some kind of heroic, altruistic act, but I think fewpeople would agree. Full Disclosure is a common practice amongst security professionals whenthe upstream project is unwilling to rectify the problem. So yes I do think the practice of Full Disclosure is an altruistic act and often times the only thing that gets people who don't care to pull their head out of the sand and actually care. You don't happen to be a random security professional, you are actually part of that upstream project and you have access to non-public (possibly confidential) data about its infrastructure, which gives you responsibilities towards your peers. I don't think I would be the only one to be angry if an infrastructure member starting publishing working exploits for unfixed vulnerabilities in the pdo infrastructure. It is a completely irresponsible way to act when you are part of a project or community. Regards Antoine. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] a plea for backward-compatibility / smooth transitions
On Mon, Jul 29, 2013 at 8:17 PM, Donald Stufft don...@stufft.io wrote: - - Distribute / setuptools merge, e.g. cratering folks who use a distro-managed 'python-distribute' package. This is the biggest issue. I wasn't involved and it could have been handled better sure. I'm not convinced. The use-cases and ways these things can be installed in conjunction with things like system packages, virtualenv and buildout in various configurations and various python versions are mindboggling. Some of them were not initially tested, probably mostly because nobody realized the finer details of what would happen in these sometimes fairly obscure cases. The problems have, afaik, now been fixed, and this was done reasonably quickly IMO. OK, of course it *could* have been handled better, but I don't think it was handled badly. The current situation is a huge mess. Untangling it *will* be tricky. //Lennart ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] a plea for backward-compatibility / smooth transitions
On Jul 30, 2013, at 5:57 AM, Antoine Pitrou solip...@pitrou.net wrote: Donald Stufft donald at stufft.io writes: On Jul 30, 2013, at 3:01 AM, Antoine Pitrou solipsis at pitrou.net wrote: I don't know what I'm supposed to infer from such a statement, except that Iprobably don't want to trust you. You might think that publish[ing] workingexploits into the wild is some kind of heroic, altruistic act, but I think fewpeople would agree. Full Disclosure is a common practice amongst security professionals whenthe upstream project is unwilling to rectify the problem. So yes I do think the practice of Full Disclosure is an altruistic act and often times the only thing that gets people who don't care to pull their head out of the sand and actually care. You don't happen to be a random security professional, you are actually part of that upstream project and you have access to non-public (possibly confidential) data about its infrastructure, which gives you responsibilities towards your peers. I don't think I would be the only one to be angry if an infrastructure member starting publishing working exploits for unfixed vulnerabilities in the pdo infrastructure. It is a completely irresponsible way to act when you are part of a project or community. I don't really care if you'd be angry. The point of Full Disclosure (and it's cousin Responsible Disclosure) is to A) Inform everyone involved that they are taking a huge risk by using a particular thing and B) Provide incentive to people to fix their shit. If others are preventing fixes from landing then both reasons still apply wether the reporter is involved with the project or not. If I can find a vulnerability then so can someone else. Someone who won't inform people and will use it to maliciously attack people. If you feel I'd be overstepping my bounds then complain to my superiors, Richard/Nick on the packaging side of things and Noah on the Infrastructure team side of things. I'll continue to do what I feel best serves the community for as long as I have the ability to do so. Which I believe is work on improving these issues, fight and advocate for the important ones, accept defeat on less important ones, and, if necessary, issue a Full Disclosure. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] a plea for backward-compatibility / smooth transitions
Donald Stufft donald at stufft.io writes: You don't happen to be a random security professional, you are actually part of that upstream project and you have access to non-public (possibly confidential) data about its infrastructure, which gives you responsibilities towards your peers. I don't think I would be the only one to be angry if an infrastructure member starting publishing working exploits for unfixed vulnerabilities in the pdo infrastructure. It is a completely irresponsible way to act when you are part of a project or community. I don't really care if you'd be angry. Great to hear. This mindset is typical of many security specialists: you're ready to tell everyone to go f*** themselves (I don't know how to voice this differently) if you think you have a higher mission to denounce some vulnerability. The point of Full Disclosure (and it's cousin Responsible Disclosure) is to A) Inform everyone involved that they are taking a huge risk by using a particular thing and B) Provide incentive to people to fix their shit. This does not necessarily involve publishing working exploits. By giving out code that can immediately attack and bring down the pdo infrastructure, you would be doing something else than merely informing the public. (neither Bruce Schneier nor Wikipedia states that Full Disclosure implies publishing working exploits, btw. I suppose it means there is at the minimum some contention, rather than consensus, over the issue.) If I can find a vulnerability then so can someone else. You may (and probably do) have domain knowledge and inside knowledge that others don't. If you feel I'd be overstepping my bounds then complain to my superiors, Richard/Nick on the packaging side of things and Noah on the Infrastructure team side of things. Superiors? Are you making things up, or do you have an org chart to back that up? :-) (regardless, I would be surprised if any of those ordered *you* to publish an exploit, rather than take the responsibility of doing it themselves - or, rather, not doing it at all) Regards Antoine. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] a plea for backward-compatibility / smooth transitions
On Tue, Jul 30, 2013 at 8:43 AM, Donald Stufft don...@stufft.io wrote: I have zero qualms about releasing a full disclosure along with working exploits into the wild for a security vulnerability that people block me on. This is a moot point, as nobody is going to block anyone. The discussion is silly. //Lennart ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] a plea for backward-compatibility / smooth transitions
On Jul 30, 2013, at 7:40 AM, Antoine Pitrou solip...@pitrou.net wrote: Donald Stufft donald at stufft.io writes: You don't happen to be a random security professional, you are actually part of that upstream project and you have access to non-public (possibly confidential) data about its infrastructure, which gives you responsibilities towards your peers. I don't think I would be the only one to be angry if an infrastructure member starting publishing working exploits for unfixed vulnerabilities in the pdo infrastructure. It is a completely irresponsible way to act when you are part of a project or community. I don't really care if you'd be angry. Great to hear. This mindset is typical of many security specialists: you're ready to tell everyone to go f*** themselves (I don't know how to voice this differently) if you think you have a higher mission to denounce some vulnerability. Keeping an issue secret doesn't make people more secure, it only prevents them from making an informed decision about the things they use. The point of Full Disclosure (and it's cousin Responsible Disclosure) is to A) Inform everyone involved that they are taking a huge risk by using a particular thing and B) Provide incentive to people to fix their shit. This does not necessarily involve publishing working exploits. By giving out code that can immediately attack and bring down the pdo infrastructure, you would be doing something else than merely informing the public. (neither Bruce Schneier nor Wikipedia states that Full Disclosure implies publishing working exploits, btw. I suppose it means there is at the minimum some contention, rather than consensus, over the issue.) Partial disclosure is typically publishing enough information to allow people to make an informed choice but (hopefully) not enough information to allow others to attack it. Full Disclosure implies laying out the full details of the vulnerability. It doesn't necessarily mean wrapping it up in ready to run code but they almost always provide enough information so that anyone with a cursory understanding of programming (or the area it affects depending on how hard it is to replicate) can implement the attack and reproduce it. The problem with Partial Disclosure is that often times vendors/upstream/whatever will simply call the problem theoretical and still attempt to not fix things. If I can find a vulnerability then so can someone else. You may (and probably do) have domain knowledge and inside knowledge that others don't. There's not really any inside knowledge. Everything is OSS. Domain knowledge? Sure, but I really hope we aren't relying on people not knowing enough about how the tooling works to exploit it. If you feel I'd be overstepping my bounds then complain to my superiors, Richard/Nick on the packaging side of things and Noah on the Infrastructure team side of things. Superiors? Are you making things up, or do you have an org chart to back that up? :-) (regardless, I would be surprised if any of those ordered *you* to publish an exploit, rather than take the responsibility of doing it themselves - or, rather, not doing it at all) Nick is the BDFL delegate for packaging tooling and Richard is the same for PyPI and I generally consider my involvement as a PyPI admin to be under him. Noah is in charge of the Infra team of which I am a member. Regards Antoine. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] a plea for backward-compatibility / smooth transitions
On Jul 30, 2013, at 7:49 AM, Lennart Regebro rege...@gmail.com wrote: On Tue, Jul 30, 2013 at 8:43 AM, Donald Stufft don...@stufft.io wrote: I have zero qualms about releasing a full disclosure along with working exploits into the wild for a security vulnerability that people block me on. This is a moot point, as nobody is going to block anyone. The discussion is silly. //Lennart *Shrug* If someone says people who don't care shouldn't have to suffer the consequences i assume that means they are advocating for no breakages in API (else the hypothetical person would be suffering). If you can't break fundamentally insecure things then that's a blockade on fixing the system. If that's not what the comment was proposing then I have no idea what it means. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] a plea for backward-compatibility / smooth transitions
On 30 Jul 2013 21:41, Antoine Pitrou solip...@pitrou.net wrote: Donald Stufft donald at stufft.io writes: You don't happen to be a random security professional, you are actually part of that upstream project and you have access to non-public (possibly confidential) data about its infrastructure, which gives you responsibilities towards your peers. I don't think I would be the only one to be angry if an infrastructure member starting publishing working exploits for unfixed vulnerabilities in the pdo infrastructure. It is a completely irresponsible way to act when you are part of a project or community. I don't really care if you'd be angry. Great to hear. This mindset is typical of many security specialists: you're ready to tell everyone to go f*** themselves (I don't know how to voice this differently) if you think you have a higher mission to denounce some vulnerability. The point of Full Disclosure (and it's cousin Responsible Disclosure) is to A) Inform everyone involved that they are taking a huge risk by using a particular thing and B) Provide incentive to people to fix their shit. This does not necessarily involve publishing working exploits. By giving out code that can immediately attack and bring down the pdo infrastructure, you would be doing something else than merely informing the public. (neither Bruce Schneier nor Wikipedia states that Full Disclosure implies publishing working exploits, btw. I suppose it means there is at the minimum some contention, rather than consensus, over the issue.) If I can find a vulnerability then so can someone else. You may (and probably do) have domain knowledge and inside knowledge that others don't. If you feel I'd be overstepping my bounds then complain to my superiors, Richard/Nick on the packaging side of things and Noah on the Infrastructure team side of things. Superiors? Are you making things up, or do you have an org chart to back that up? :-) Effectively he does, yes. Richard is responsible for approving PyPI API changes (including PyPI specific PEPs), I'm BDFL delegate for other packaging PEPs and Noah has final say over the operation of the python.orginfrastructure. One or more of us are the ones that need to say yes on potentially controversial changes, so the responsibility for any mistakes ultimately lies with us, rather than Donald (and I'm greatly appreciative of the huge amount of work he is putting into improving the PyPI security story). (regardless, I would be surprised if any of those ordered *you* to publish an exploit, rather than take the responsibility of doing it themselves - or, rather, not doing it at all) If Donald informed us of a vulnerability and we refused to allow him (or anyone else) to take the necessary steps to close it, then he would be *completely* justified in publishing full details of the vulnerability, up to and including working exploit code. It won't come to that though, because we're taking this seriously and closing security holes as quickly as is feasible while still ensuring a reasonable level of backwards compatibility :) Cheers, Nick. Regards Antoine. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] a plea for backward-compatibility / smooth transitions
On Jul 30, 2013, at 8:06 AM, Nick Coghlan ncogh...@gmail.com wrote: If Donald informed us of a vulnerability and we refused to allow him (or anyone else) to take the necessary steps to close it, then he would be *completely* justified in publishing full details of the vulnerability, up to and including working exploit code. It won't come to that though, because we're taking this seriously and closing security holes as quickly as is feasible while still ensuring a reasonable level of backwards compatibility :) This basically. Maybe I'm not being clear because I have a headache and I'm reading too much into things because I'm sensitive to being shutdown on efforts to fix these things*. I don't expect with Nick, Richard, and Noah to ever need to do a Full Disclosure. I was only trying to be clear about what I consider my escalation path to be if a current, or near future vulnerability is forced to remain open. * I started trying to push for this ~2 years ago and got repeatedly shut down, for one reason or another. Which lead to to create Crate.io. It's only been relatively recently that I've been given permission to actually fix things. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] a plea for backward-compatibility / smooth transitions
On Tue, Jul 30, 2013 at 4:14 AM, Donald Stufft don...@stufft.io wrote: Heh, I'm pretty good at getting yelled at :) Nick is also pretty good at making people feel like he both knows and *cares* about their breakage, and isn't just dismissing their concerns as trivial or unimportant. Breakage isn't trivial or unimportant to the person who's yelling, so this is an important community-maintenance skill. It builds trust, and reduces the total amount of yelling. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] a plea for backward-compatibility / smooth transitions
On Jul 30, 2013, at 1:13 PM, PJ Eby p...@telecommunity.com wrote: On Tue, Jul 30, 2013 at 4:14 AM, Donald Stufft don...@stufft.io wrote: Heh, I'm pretty good at getting yelled at :) Nick is also pretty good at making people feel like he both knows and *cares* about their breakage, and isn't just dismissing their concerns as trivial or unimportant. Breakage isn't trivial or unimportant to the person who's yelling, so this is an important community-maintenance skill. It builds trust, and reduces the total amount of yelling. *shrug*, If I didn't care I would have made this change as soon as Nick said it was ok. Instead I declared I was going to and waited to make sure nobody else had any concerns. And once Holger said he did I said ok I won't do it. Maybe my mannerisms give the impression I don't but that's actually pretty far from the truth. For this particular change I originally created the pip commit that allowed it, and then again I created the setuptools commit, backporting hashlib into setuptools to support Python 2.4. I put a decent amount of effort into trying to make sure that nothing broke but in the end there were still concerns :) - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] a plea for backward-compatibility / smooth transitions
On Tue, Jul 30, 2013 at 2:04 PM, Donald Stufft don...@stufft.io wrote: On Jul 30, 2013, at 1:13 PM, PJ Eby p...@telecommunity.com wrote: On Tue, Jul 30, 2013 at 4:14 AM, Donald Stufft don...@stufft.io wrote: Heh, I'm pretty good at getting yelled at :) Nick is also pretty good at making people feel like he both knows and *cares* about their breakage, and isn't just dismissing their concerns as trivial or unimportant. Breakage isn't trivial or unimportant to the person who's yelling, so this is an important community-maintenance skill. It builds trust, and reduces the total amount of yelling. *shrug*, If I didn't care I would have made this change as soon as Nick said it was ok. Instead I declared I was going to and waited to make sure nobody else had any concerns. And once Holger said he did I said ok I won't do it. Maybe my mannerisms give the impression I don't but that's actually pretty far from the truth. I did say feel like. ;-) Nick usually gives more of an impression that he's thought about concerns raised before rejecting them; your responses often sound like, Who cares about that? Asking for suggestions, for example, would be good. Nick also rarely seems irritated by people's concerns or problems, whereas you sometimes seem in a big hurry to fix something today or this week that's been broken for years, without giving folks a while to get used to the idea. Often your proposals seem less like proposals than, I've decided to do this, so deal with it. I'm not saying all this because I want to complain or yell at you; I'm saying it because I think you do care enough to know how you're coming across, at least to me. Our discussions have gotten heated in the past because my impression of your reaction to the concerns I raise often seems like, I don't care about supporting [group of people affected], so neither should you. Perhaps the issue is just one of confusion. When I raise an issue about, say, Python 2.3 users (who are still downloading setuptools 0.6 releases, and presumably also using them), it's not because I expect *you* to change your plans to support them, but because I need to know how *I* can, if the issue arises. So I don't actually expect you to care about Python 2.3 users (again, as an example), but I do expect you to care about *me* supporting them. In the most recent situation, you did in fact point me to your awesome hashlib port, so I do know you *do* care to at least that extent. But the rhetoric that you sent both before and after the helpful bit seemed on the inflammatory side, as though I were crazy to be thinking of Python 2.3. Whether or not this is true ;-) -- it's not especially *helpful* to the discussion. If I may offer a suggestion, asking questions in response to objections is generally more useful than immediately arguing the relevance of the objection. First, it tells the objector that you're interested in what they have to say, and second, it may well help the objector understand that there isn't actually any real problem, and gives them an easier path to backing down and/or compromising, whereas a frontal assault tends to focus people on responding to you instead of reconsidering their objection. On the hashlib issue, for example, it actually occurred to me later that it's completely a non-issue because the actual hash scenario I was most concerned about *can't actually happen*. (MD5 hashes in code or dependency_links, used e.g. by setuptools itself to secure its own downloads. Changing PyPI won't affect these, duh.) It might've occurred to me sooner, though, if you'd actually asked what scenario I was worried about, instead of arguing about the relevance. This isn't to say that you're responsible for what I do or don't figure out; my point is simply that asking questions and inviting suggestions in response to people's objections will generally get you more thoughtful responses and more trust, and resolve issues sooner, with less arguing. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] a plea for backward-compatibility / smooth transitions
On Jul 30, 2013, at 4:01 PM, PJ Eby p...@telecommunity.com wrote: On Tue, Jul 30, 2013 at 2:04 PM, Donald Stufft don...@stufft.io wrote: On Jul 30, 2013, at 1:13 PM, PJ Eby p...@telecommunity.com wrote: On Tue, Jul 30, 2013 at 4:14 AM, Donald Stufft don...@stufft.io wrote: Heh, I'm pretty good at getting yelled at :) Nick is also pretty good at making people feel like he both knows and *cares* about their breakage, and isn't just dismissing their concerns as trivial or unimportant. Breakage isn't trivial or unimportant to the person who's yelling, so this is an important community-maintenance skill. It builds trust, and reduces the total amount of yelling. *shrug*, If I didn't care I would have made this change as soon as Nick said it was ok. Instead I declared I was going to and waited to make sure nobody else had any concerns. And once Holger said he did I said ok I won't do it. Maybe my mannerisms give the impression I don't but that's actually pretty far from the truth. I did say feel like. ;-) Nick usually gives more of an impression that he's thought about concerns raised before rejecting them; your responses often sound like, Who cares about that? Asking for suggestions, for example, would be good. Nick also rarely seems irritated by people's concerns or problems, whereas you sometimes seem in a big hurry to fix something today or this week that's been broken for years, without giving folks a while to get used to the idea. Often your proposals seem less like proposals than, I've decided to do this, so deal with it. I'm not saying all this because I want to complain or yell at you; I'm saying it because I think you do care enough to know how you're coming across, at least to me. Our discussions have gotten heated in the past because my impression of your reaction to the concerns I raise often seems like, I don't care about supporting [group of people affected], so neither should you. Perhaps the issue is just one of confusion. When I raise an issue about, say, Python 2.3 users (who are still downloading setuptools 0.6 releases, and presumably also using them), it's not because I expect *you* to change your plans to support them, but because I need to know how *I* can, if the issue arises. So I don't actually expect you to care about Python 2.3 users (again, as an example), but I do expect you to care about *me* supporting them. In the most recent situation, you did in fact point me to your awesome hashlib port, so I do know you *do* care to at least that extent. But the rhetoric that you sent both before and after the helpful bit seemed on the inflammatory side, as though I were crazy to be thinking of Python 2.3. Whether or not this is true ;-) -- it's not especially *helpful* to the discussion. If I may offer a suggestion, asking questions in response to objections is generally more useful than immediately arguing the relevance of the objection. First, it tells the objector that you're interested in what they have to say, and second, it may well help the objector understand that there isn't actually any real problem, and gives them an easier path to backing down and/or compromising, whereas a frontal assault tends to focus people on responding to you instead of reconsidering their objection. On the hashlib issue, for example, it actually occurred to me later that it's completely a non-issue because the actual hash scenario I was most concerned about *can't actually happen*. (MD5 hashes in code or dependency_links, used e.g. by setuptools itself to secure its own downloads. Changing PyPI won't affect these, duh.) It might've occurred to me sooner, though, if you'd actually asked what scenario I was worried about, instead of arguing about the relevance. This isn't to say that you're responsible for what I do or don't figure out; my point is simply that asking questions and inviting suggestions in response to people's objections will generally get you more thoughtful responses and more trust, and resolve issues sooner, with less arguing. Hrm. So I hear what you're saying and part of the problem is likely due to the history of where I tried to get a change through and then felt like all I was getting was stop energy and people wanting to keep the status quo which ultimately ended up preventing changes has lead me to view distutils-sig in more of an adversarial light then is probably appropriate for the distutils-sig of 2013 (versus the distutils-sig of 2011/2012). This is probably reflected in my tone and likely has others, such as yourself, respond similarly, pushing us further down that path. My thought process has become Ok here's something that needs to happen, now how do I get distutils-sig not to prevent it from happening. This was again reflected in the Python 2.3 discussion because my immediate reaction and impression was that you were attempting to block the move from MD5 due to
Re: [Distutils] a plea for backward-compatibility / smooth transitions
On 30 July 2013 21:58, Donald Stufft don...@stufft.io wrote: As far as how to fix it I don't have a particularly magic answer. I will try to be more mindful of my tone and that distutils-sig is likely not my adversary anymore as well as try to ask questions instead of arguing the relevance immediately. Thank you - both for the thoughtful response and the explanation of what is going on in your thinking. I certainly tend to think about possible issues rather than benefits, maybe more than I should as I don't personally have any particular problems with a rapid pace of change (except in some specific areas where I'm in so much of a minority that I don't expect my concerns to be a big driver in the grand scheme of things). I think I'm doing this to make sure people haven't missed potential issues I'm aware of, but I can easily imagine that this comes across as negative stop energy. I'll try to stick to issues where I have genuine concerns, and not theoretical ones in future. And yes, it does feel like distutils-sig of 2013 is a much nicer place than it used to be. Thanks to everyone for working to keep that the case. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] a plea for backward-compatibility / smooth transitions
Donald Stufft donald at stufft.io writes: to PyPI. The snark in my statements primarily came from that feeling of someone was trying to shut down an enhancement. To be fair, the tone you've taken (which can rub people up the wrong way) hasn't just been over long-standing security issues. I've also felt uncomfortable with some of your responses relating to the PEP 426 discussion around the JSON schema for requirements, where there is not the same level of urgency, nor valid reason to believe that you're being thwarted somehow. If someone's approach comes off as unduly dismissive (even if not meant that way), it makes it harder to engage in constructive discussion. I recognise that it's hard communicating with people remotely, many of whom one might never have met. It's easy to miss nuance and misconstrue turns of phrase. It's best to tread gingerly, else too much time is spent smoothing ruffled feathers at the expense of getting real work done. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] a plea for backward-compatibility / smooth transitions
On Jul 30, 2013, at 5:17 PM, Paul Moore p.f.mo...@gmail.com wrote: On 30 July 2013 21:58, Donald Stufft don...@stufft.io wrote: As far as how to fix it I don't have a particularly magic answer. I will try to be more mindful of my tone and that distutils-sig is likely not my adversary anymore as well as try to ask questions instead of arguing the relevance immediately. Thank you - both for the thoughtful response and the explanation of what is going on in your thinking. I certainly tend to think about possible issues rather than benefits, maybe more than I should as I don't personally have any particular problems with a rapid pace of change (except in some specific areas where I'm in so much of a minority that I don't expect my concerns to be a big driver in the grand scheme of things). I think I'm doing this to make sure people haven't missed potential issues I'm aware of, but I can easily imagine that this comes across as negative stop energy. I'll try to stick to issues where I have genuine concerns, and not theoretical ones in future. I don't actually mind even theoretical problems nor do I want people to feel like they need to coddle me because of my history with distutils-sig. That history and how It affects me is *my* problem. I think as long as we, including myself, approach theoretical problems as Let's figure out if this theoretical problem is actually a problem, and if it is do we care about it and not here's a bunch of possible problems, we can't do this then there will be no issues. It's true that theoretical problems can make me feel more like someones applying stop energy than concrete problems, but that's not a problem of the person bringing up that problem and more just it triggering old feelings from a time where I couldn't even get a SSL certificate trusted by most major browsers by default deployed ;). Not allowing those feelings to poison current efforts is on me. And yes, it does feel like distutils-sig of 2013 is a much nicer place than it used to be. Thanks to everyone for working to keep that the case. Paul - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] a plea for backward-compatibility / smooth transitions
On Jul 30, 2013, at 6:19 PM, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: Donald Stufft donald at stufft.io writes: to PyPI. The snark in my statements primarily came from that feeling of someone was trying to shut down an enhancement. To be fair, the tone you've taken (which can rub people up the wrong way) hasn't just been over long-standing security issues. I've also felt uncomfortable with some of your responses relating to the PEP 426 discussion around the JSON schema for requirements, where there is not the same level of urgency, nor valid reason to believe that you're being thwarted somehow. I never meant to claim it was simply over security issues. I'm certainly more energetic about advocating for security issues and they frame the backdrop for a lot of my feelings with distutils-sig as an adversary rather than an ally . The simple fact is that every disagreement, or even simple suggestion, is framed against that feeling leading towards the Me vs Them tone. If someone's approach comes off as unduly dismissive (even if not meant that way), it makes it harder to engage in constructive discussion. I recognise that it's hard communicating with people remotely, many of whom one might never have met. It's easy to miss nuance and misconstrue turns of phrase. It's best to tread gingerly, else too much time is spent smoothing ruffled feathers at the expense of getting real work done. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] a plea for backward-compatibility / smooth transitions (was: Re: Migrating Hashes from MD5 to SHA256)
Hi Nick, Donald, all, On Sun, Jul 28, 2013 at 22:23 +1000, Nick Coghlan wrote: On 28 July 2013 20:55, Donald Stufft don...@stufft.io wrote: Ok so given that: - There's a readably available solution for Python 2.4+ with the likelihood being that most users are either using it or using an older version which doesn't support SSL. - The number of folks likely to be on Python 2.3 and wanting to install things from PyPI is likely to be very small. - There's possibly a future solution for Python 2.3 - The safety margins for MD5 are gone and cryptographers heavily suggest moving away from it. Please detail the actual attack scenario wrt PyPI/installer processes. - A revised scheme will break backwards compatibility with the versions of the tooling that do support a stronger hash. I'm going to go ahead and make this change unless someone comes out and contests moving PyPI to SHA256. I'll give it a bit to make sure no one does have an issue with the move. Actually, i strongly object further backward-incompatible changes. Please (generally) find a way to introduce improvements without breaking existing installation processes at the same time. For example, in this case pip/easy_install could indicate to PYPI what kind of hashes it accepts (through a header or query param or whatever) and PyPI could serve it but we'd default to MD5 for now if nothing else was requested. Please also consider the PEP438 vetted registration of externals+hashses in this context. Once things and tools are working nicely we can switch to serving a non-MD5 hash as default after a sufficient grace period. +1, this sounds like a good way forward for the existing PyPI interfaces. We can do something better once the focus shifts from make the status quo not broken to making the next generation interfaces a reality (PEP 426 et al). Just switching the hashes is likely to break things. Do you want to take a bet on that? Just switching to SSL broke things. Just switching pip-1.4 broke things. Just switching to newer setuptools broke things. For me the fall-out of all the well-intentioned changes has been frustrating lately. For example, one of my projects documented how to generate a supervisor-controled deployment and suddenly pip install supervisor does not work anymore because pip-1.4 doesn't consider any existing supervisor distributions as non-dev. You can argue that's supervisor's fault but in the end it doesn't matter: What used to work and was documented is now broken, users complaining or simply turning away, with a re-inforced view of python packaging sucks or this tool sucks, the docs are wrong. There are 1000 times more people using python packaging tools than there are ones following the recent distutils-sig/Donald/Nick verdicts or tool release notes. And heck, even for me being quite involved in all this stuff, having written a related PEP, it's really hard to see what's coming and to prevent breakage for my tools's/projects's users. Therefore, to be honest, at this point i am almost afraid of what is released/deployed next in PyPI/Installer land. I'd rather like to get into a can't wait to get the next release mode. best, holger ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] a plea for backward-compatibility / smooth transitions (was: Re: Migrating Hashes from MD5 to SHA256)
On 29 July 2013 19:38, holger krekel hol...@merlinux.eu wrote: Hi Nick, Donald, all, On Sun, Jul 28, 2013 at 22:23 +1000, Nick Coghlan wrote: On 28 July 2013 20:55, Donald Stufft don...@stufft.io wrote: Ok so given that: - There's a readably available solution for Python 2.4+ with the likelihood being that most users are either using it or using an older version which doesn't support SSL. - The number of folks likely to be on Python 2.3 and wanting to install things from PyPI is likely to be very small. - There's possibly a future solution for Python 2.3 - The safety margins for MD5 are gone and cryptographers heavily suggest moving away from it. Please detail the actual attack scenario wrt PyPI/installer processes. - A revised scheme will break backwards compatibility with the versions of the tooling that do support a stronger hash. I'm going to go ahead and make this change unless someone comes out and contests moving PyPI to SHA256. I'll give it a bit to make sure no one does have an issue with the move. Actually, i strongly object further backward-incompatible changes. Please (generally) find a way to introduce improvements without breaking existing installation processes at the same time. For example, in this case pip/easy_install could indicate to PYPI what kind of hashes it accepts (through a header or query param or whatever) and PyPI could serve it but we'd default to MD5 for now if nothing else was requested. Please also consider the PEP438 vetted registration of externals+hashses in this context. Once things and tools are working nicely we can switch to serving a non-MD5 hash as default after a sufficient grace period. Having the improved hashes be opt-in (by the client) strikes me as a reasonable request. Yes, this means nothing will actually happen until easy_install/pip are updated to request those improved hashes and those versions see significant uptake, but as Holger says, we need to ensure we put sufficient effort into smoothing out the roller coaster ride that has been the recent experience of packaging system users. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] a plea for backward-compatibility / smooth transitions (was: Re: Migrating Hashes from MD5 to SHA256)
On Jul 29, 2013, at 7:58 AM, Nick Coghlan ncogh...@gmail.com wrote: Actually, i strongly object further backward-incompatible changes. Please (generally) find a way to introduce improvements without breaking existing installation processes at the same time. For example, in this case pip/easy_install could indicate to PYPI what kind of hashes it accepts (through a header or query param or whatever) and PyPI could serve it but we'd default to MD5 for now if nothing else was requested. Please also consider the PEP438 vetted registration of externals+hashses in this context. Once things and tools are working nicely we can switch to serving a non-MD5 hash as default after a sufficient grace period. Having the improved hashes be opt-in (by the client) strikes me as a reasonable request. Yes, this means nothing will actually happen until easy_install/pip are updated to request those improved hashes and those versions see significant uptake, but as Holger says, we need to ensure we put sufficient effort into smoothing out the roller coaster ride that has been the recent experience of packaging system users. There's basically zero way for this to fail closed in any of the current installers. The failure mode is unverified packages not uninstallable packages. I am not aware of a single installer that mandates the use of a hash. Crate.io has never used md5 hashes and has always used sha256 and I've never received a single report of an installer being unable to install because of it, which is exactly what I expect. Indicating via Header or query param pretty much destroys the effectiveness of the CDN's cache in order to fix a problem with a theoretical (as far as I am aware) installer that requires a md5 hash (and thus has never worked for any of the externally hosted packages. Additionally it doesn't account for external urls which need to be registered *with* a hash. As far as available attacks, *today* an author could upload a package that has been created so as to have a sister package with malicious code that has the same hash allowing them to have a malicious package they can substitute at will without the hashes changing at all. In the future it's possible that a pre-image attack on MD5 will be found and then we'll be dealing with this problem then when we've lost all verification on external urls instead of now when we have time to get external urls to switch. So by all means I will not migrate us if that's what you want. Old versions of the installation clients stick around far to long for the opt in mechanism to be much use. The point of switching was to cover the existing clients as well to narrow the gap until a new API is developed. Hopefully no one is relying on these hashes to prevent an author from maliciously injecting a sister package and hopefully the strength of MD5 holds and no new research is found that blows it's pre-image attack residence to pieces. As far as not breaking things goes backwards compatibility has been an important concern however progress forward *requires* breakage. It is required because there is a vast array of available ways to have your package and/or hosting configured many of them horrible practices which need to be killed. Killing them requires breaking backwards compatibility. You cite SSL, yes SSL has caused a number of errors for people mostly related to older versions of OpenSSL being unable to use a SSL certificate but downloading code you're going to execute over plaintext isn't just bad, it's downright negligent on the part of the toolchain. So that was a required breakage. You also mention the pip 1.4 *not* installing pre-releases by default. Yes that broke a handful of packages Supervisor and pytz being the major ones that I've seen anyone complain about. It was also known ahead of time that this was a backwards incompatible change (and it was noted as such in the release notes). It wasn't a surprising outcome. The pip developers drew a line in the sand to quote Paul Moore and I expect pip 1.5 where PEP438 becomes default to break even more packages from people who just haven't bothered to change their practices until it's forced on them. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] a plea for backward-compatibility / smooth transitions
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 07/29/2013 10:51 AM, Donald Stufft wrote: I've personally gotten or seen more complaints over the naming of a variable in the config file then I have over any changes we've made. The runner up to that is the fallout from switching to requiring verified SSL. The past few months have generated a *lot* of teeth-gnashing / hair-pulling, especially among downstream developers (those unlikely to be reading this SIG): - - HTTPS-only PyPY - - Distribute / setuptools merge, e.g. cratering folks who use a distro-managed 'python-distribute' package. - - Pip's new backward-imcompatible final releases by default. I think we are going to be in a much better place for all that, but let's not deny the pain involved for *everybody* in getting there. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlH2n+EACgkQ+gerLs4ltQ6megCeN8V8IN4OT6rWfZg1tw1GtUaO 2jwAoLRXzLyjjMbiLrcfqLG0Ge1NQnMq =7ufm -END PGP SIGNATURE- ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] a plea for backward-compatibility / smooth transitions
On 29 July 2013 18:01, Tres Seaver tsea...@palladion.com wrote: I think we are going to be in a much better place for all that, but let's not deny the pain involved for *everybody* in getting there. Agreed. I think the goal is valid, and the approach is fine. But we need to do a better job in managing people's expectations. I'd like to see a roadmap of the various changes planned, as well as some sort of explanation of how each of the changes contributes towards the end goal. Personally, none of the changes have detrimentally affected me, so my opinion is largely theoretical. But even I am getting a little frustrated by the constant claims that what we have now is insecure and broken, and must be fixed ASAP. The reality is that everything's more or less OK - there's a risk, certainly, and it could be severe, but many, many people are routinely using PyPI all the time without issues. And telling them that they are wrong to do so, or that they are being extremely naive over security, isn't helping. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] a plea for backward-compatibility / smooth transitions (was: Re: Migrating Hashes from MD5 to SHA256)
On Mon, Jul 29, 2013 at 10:30 -0400, Donald Stufft wrote: On Jul 29, 2013, at 7:58 AM, Nick Coghlan ncogh...@gmail.com wrote: Actually, i strongly object further backward-incompatible changes. Please (generally) find a way to introduce improvements without breaking existing installation processes at the same time. For example, in this case pip/easy_install could indicate to PYPI what kind of hashes it accepts (through a header or query param or whatever) and PyPI could serve it but we'd default to MD5 for now if nothing else was requested. Please also consider the PEP438 vetted registration of externals+hashses in this context. Once things and tools are working nicely we can switch to serving a non-MD5 hash as default after a sufficient grace period. Having the improved hashes be opt-in (by the client) strikes me as a reasonable request. Yes, this means nothing will actually happen until easy_install/pip are updated to request those improved hashes and those versions see significant uptake, but as Holger says, we need to ensure we put sufficient effort into smoothing out the roller coaster ride that has been the recent experience of packaging system users. There's basically zero way for this to fail closed in any of the current installers. The failure mode is unverified packages not uninstallable packages. I am not aware of a single installer that mandates the use of a hash. Crate.io has never used md5 hashes and has always used sha256 and I've never received a single report of an installer being unable to install because of it, which is exactly what I expect. So you think the worst case for forcing SHA256 hashes is that installers who don't yet support sha256 hashes would just ignore it (and thus wouldn't do hash verification)? Indicating via Header or query param pretty much destroys the effectiveness of the CDN's cache in order to fix a problem with a theoretical (as far as I am aware) installer that requires a md5 hash (and thus has never worked for any of the externally hosted packages. Additionally it doesn't account for external urls which need to be registered *with* a hash. Currently there is no hash-type enforcement on registered externals, is there? As far as available attacks, *today* an author could upload a package that has been created so as to have a sister package with malicious code that has the same hash allowing them to have a malicious package they can substitute at will without the hashes changing at all. In the future it's possible that a pre-image attack on MD5 will be found and then we'll be dealing with this problem then when we've lost all verification on external urls instead of now when we have time to get external urls to switch. So the attack is a malicious author or someone else modifying an external release file (either directly on the server or via MITM) while maintaining the pre-registered MD5 hash, right? I am currently merely trying to understand more exactly what you are worried about. best, holger So by all means I will not migrate us if that's what you want. Old versions of the installation clients stick around far to long for the opt in mechanism to be much use. The point of switching was to cover the existing clients as well to narrow the gap until a new API is developed. Hopefully no one is relying on these hashes to prevent an author from maliciously injecting a sister package and hopefully the strength of MD5 holds and no new research is found that blows it's pre-image attack residence to pieces. As far as not breaking things goes backwards compatibility has been an important concern however progress forward *requires* breakage. It is required because there is a vast array of available ways to have your package and/or hosting configured many of them horrible practices which need to be killed. Killing them requires breaking backwards compatibility. You cite SSL, yes SSL has caused a number of errors for people mostly related to older versions of OpenSSL being unable to use a SSL certificate but downloading code you're going to execute over plaintext isn't just bad, it's downright negligent on the part of the toolchain. So that was a required breakage. You also mention the pip 1.4 *not* installing pre-releases by default. Yes that broke a handful of packages Supervisor and pytz being the major ones that I've seen anyone complain about. It was also known ahead of time that this was a backwards incompatible change (and it was noted as such in the release notes). It wasn't a surprising outcome. The pip developers drew a line in the sand to quote Paul Moore and I expect pip 1.5 where PEP438 becomes default to break even more packages from people who just haven't bothered to change their practices until it's forced on them. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Re: [Distutils] a plea for backward-compatibility / smooth transitions
On Mon, Jul 29, 2013 at 10:18 AM, Paul Moore p.f.mo...@gmail.com wrote: On 29 July 2013 18:01, Tres Seaver tsea...@palladion.com wrote: I think we are going to be in a much better place for all that, but let's not deny the pain involved for *everybody* in getting there. Agreed. I think the goal is valid, and the approach is fine. But we need to do a better job in managing people's expectations. I'd like to see a roadmap of the various changes planned, as well as some sort of explanation of how each of the changes contributes towards the end goal. Nick said he's planning a roadmap on the new User Guide (in August I think after vacation), so that's something. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] a plea for backward-compatibility / smooth transitions
On Jul 29, 2013, at 1:18 PM, Paul Moore p.f.mo...@gmail.com wrote: But even I am getting a little frustrated by the constant claims that what we have now is insecure and broken, and must be fixed ASAP. The reality is that everything's more or less OK - there's a risk, certainly, and it could be severe, but many, many people are routinely using PyPI all the time without issues. And telling them that they are wrong to do so, or that they are being extremely naive over security, isn't helping. This shows a fundamental misunderstanding of how security issues present themselves. Of course things just work for people because security issues are not like regular bugs. They don't negatively affect you until someone attempts to use them to attack you. Keep your front door unlocked on your house and your valuables will remain inside _until_ someone decides to try and rob you. If you wait until people are affected by a security vulnerability then the horse has already fled the pasture and you're just attempting to close the gate after the fact. I'm pushing hard on doing what we can to secure the infrastructure because this shit matters. Everything is more or less OK, only because no one has decided that people installing from PyPI are not a valuable enough target to go after. Prior to this push that was basically the only thing prevent someone from attacking people, that they had never decided to bother too. We are better, it's somewhat harder now, but in many areas that's still the only thing keeping people safe. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] a plea for backward-compatibility / smooth transitions
On Mon, Jul 29, 2013 at 2:15 PM, Donald Stufft don...@stufft.io wrote: On Jul 29, 2013, at 1:18 PM, Paul Moore p.f.mo...@gmail.com wrote: But even I am getting a little frustrated by the constant claims that what we have now is insecure and broken, and must be fixed ASAP. The reality is that everything's more or less OK - there's a risk, certainly, and it could be severe, but many, many people are routinely using PyPI all the time without issues. And telling them that they are wrong to do so, or that they are being extremely naive over security, isn't helping. This shows a fundamental misunderstanding of how security issues present themselves. Of course things just work for people because security issues are not like regular bugs. They don't negatively affect you until someone attempts to use them to attack you. Keep your front door unlocked on your house and your valuables will remain inside _until_ someone decides to try and rob you. If you wait until people are affected by a security vulnerability then the horse has already fled the pasture and you're just attempting to close the gate after the fact. I'm pushing hard on doing what we can to secure the infrastructure because this shit matters. Everything is more or less OK, only because no one has decided that people installing from PyPI are not a valuable enough target to go after. Prior to this push that was basically the only thing prevent someone from attacking people, that they had never decided to bother too. We are better, it's somewhat harder now, but in many areas that's still the only thing keeping people safe. Well said. Security is a pain, but I'm really glad and appreciate that you and others are paying attention to it. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] a plea for backward-compatibility / smooth transitions (was: Re: Migrating Hashes from MD5 to SHA256)
On Jul 29, 2013, at 1:28 PM, holger krekel hol...@merlinux.eu wrote: On Mon, Jul 29, 2013 at 10:30 -0400, Donald Stufft wrote: On Jul 29, 2013, at 7:58 AM, Nick Coghlan ncogh...@gmail.com wrote: Actually, i strongly object further backward-incompatible changes. Please (generally) find a way to introduce improvements without breaking existing installation processes at the same time. For example, in this case pip/easy_install could indicate to PYPI what kind of hashes it accepts (through a header or query param or whatever) and PyPI could serve it but we'd default to MD5 for now if nothing else was requested. Please also consider the PEP438 vetted registration of externals+hashses in this context. Once things and tools are working nicely we can switch to serving a non-MD5 hash as default after a sufficient grace period. Having the improved hashes be opt-in (by the client) strikes me as a reasonable request. Yes, this means nothing will actually happen until easy_install/pip are updated to request those improved hashes and those versions see significant uptake, but as Holger says, we need to ensure we put sufficient effort into smoothing out the roller coaster ride that has been the recent experience of packaging system users. There's basically zero way for this to fail closed in any of the current installers. The failure mode is unverified packages not uninstallable packages. I am not aware of a single installer that mandates the use of a hash. Crate.io has never used md5 hashes and has always used sha256 and I've never received a single report of an installer being unable to install because of it, which is exactly what I expect. So you think the worst case for forcing SHA256 hashes is that installers who don't yet support sha256 hashes would just ignore it (and thus wouldn't do hash verification)? Yes. I've been using sha256 on simple.crate.io for over a year and zero people have ever stated it didn't work for them. This also fits in with my knowledge of how setuptools and pip works. I know zc.buildout less well but to my knowledge they simple allow setuptools to handle the downloading. Indicating via Header or query param pretty much destroys the effectiveness of the CDN's cache in order to fix a problem with a theoretical (as far as I am aware) installer that requires a md5 hash (and thus has never worked for any of the externally hosted packages. Additionally it doesn't account for external urls which need to be registered *with* a hash. Currently there is no hash-type enforcement on registered externals, is there? Registered externals must register with a md5 hash, scraped links and download urls etc do not require it because they are indirectly added. There is no verification by PyPI that the given hash matches the package at the end of the url. As far as available attacks, *today* an author could upload a package that has been created so as to have a sister package with malicious code that has the same hash allowing them to have a malicious package they can substitute at will without the hashes changing at all. In the future it's possible that a pre-image attack on MD5 will be found and then we'll be dealing with this problem then when we've lost all verification on external urls instead of now when we have time to get external urls to switch. So the attack is a malicious author or someone else modifying an external release file (either directly on the server or via MITM) while maintaining the pre-registered MD5 hash, right? I am currently merely trying to understand more exactly what you are worried about. For any hash function there are two major types of attacks you worry about. The first is a collision attack which is the ability to generate two arbitrary inputs that hash to the same thing. The second is a pre-image attack (either first or second pre-image) which essentially means given an already existing input generate another input that hashes to the same thing. So basically the difference between the two attacks are wether you have a hash you're trying to match or if you just need two inputs that hash to the same thing. MD5 is currently broken for collision resistance. This means that an author can generate two packages that hash to the same thing. Once package might be benign and one might be malicious. Given those two packages people using the md5 hashes will not be able to differentiate between the benign and the malicous package. MD5 is currently *not* broken for pre image resistance. This means that as of right now someone can not take an already existing package on PyPI and generate a second package that hashes to the same thing (besides via brute forcing). So right now, collision attacks possible == yes, pre image attacks possible == no. However designing secure systems is a practice of building in safety margins. If someone, for instance, can break 5 rounds of a function you use 15 rounds. With
Re: [Distutils] a plea for backward-compatibility / smooth transitions
Folks: dstufft already correctly explained that relying on MD5 allows for doppelganger packages -- two (or more) packages which are engineered at birth to have the same hash as each other. It isn't clear to me that this can be used for evil, but it isn't obvious that it *can't* be used for evil, either. So it would certainly be helpful to upgrade the hash function so that we don't have to think about that anymore, but in my opinion it is not an emergency. I'd like to push back on the other risk, that someone might figure out how to make MD5 second-pre-images. I don't think this is a risk that we need to urgently address, and I've written a short note explaining why. This note is incomplete, badly edited, has not been peer-reviewed, and is not ready for publication, but I thought it might help folks evaluate how urgent it is to upgrade from MD5, so here it is. Regards, Zooko ??? the historical success of collision attacks does not imply a danger of pre-image attacks by Zooko Wilcox-O'Hearn, LeastAuthority.com, 2013-07-29 Summary === Most of the secure hash functions designed before 2007 have turned out to be vulnerable to collision attacks. This includes the widely-used secure hash functions MD5 and SHA-1. Newer hash functions, including those designed since the SHA-3 project began in 2007, do not appear to be vulnerable to collision attacks, but since they are newer, there has also been less time for cryptanalysts to find flaws in them. A widely cited web page shows a graphical representation of the history of various hash functions being broken: http://valerieaurora.org/hash.html The advice on that web page is that if you are relying on your hash function for collision-resistance, then you should be prepared to migrate to a new hash function every few years. .. insert table based on the main table below, but showing only vulnerability to collisions instead of pre-images What about pre-image attacks or second pre-image attacks? Some systems require their secure hash function to resist pre-image and 2nd-pre-image attacks, but do not require their secure hash function to resist collision attacks. For such systems, an interesting question is whether many or few of the secure hash functions published in the last 23 years???since the advent of modern cryptography???have turned out to be vulnerable to pre-image or 2nd-pre-image attacks. The answer is that with a single exception, no secure hash function has ever been shown to be vulnerable to (2nd-)pre-image attacks. That single exception is the second-oldest hash function ever designed, Snefru, which was designed in 1989 and 1990, and which turned out to be vulnerable to differential cryptanalysis. Differential cryptanalysis was discovered (by the open research community) in 1990. Preliminaries = The input to a secure hash function is called the *pre-image* and the output is called the *image*. A hash function *collision* is two different inputs (pre-images) which result in the same output. A hash function is *collision-resistant* if an adversary can't find any collision. A hash function is *pre-image resistant* if, given an output (image), an adversary can't find any input (pre-image) which results in that output. A hash function is *second pre-image resistant* if, given a pre-image, an adversary can't find any *other* pre-image which results in the same image. Motivation == My motivation for caring about pre-image resistance and 2nd-pre-image resistance is that it is possible to build digital signatures from secure hash functions. Some of these *hash-based digital signatures* have been proven to be secure (resistant to forgery) as long as the hash function they are built out of has second-pre-image resistance, e.g. Buchmann-2011_. Such a hash-based digital signature would fail if its underlying hash function failed at second-pre-image resistance, but this is the *only* way that it could be broken???any attack which was able to forge digital signatures against such a scheme would *have* to violate the second-pre-image resistance of the underlying hash function. One reason that hash-based digital signatures might be useful is that if an attacker has a sufficiently large quantum computer, they could forge digital signatures that rely on factorization or discrete log, such as RSA, DSA, ECDSA, or Ed25519. There is not any reason to think that such a quantum computer would enable them to break secure hash functions, however. Survey of attacks on hash functions === We know that practical, widely-deployed secure hash functions have turned out to be vulnerable to collision attacks. MD5 is the most widely used secure hash function which turns out to admit collisions, but many other secure hash functions have
Re: [Distutils] a plea for backward-compatibility / smooth transitions
On Jul 29, 2013, at 2:57 PM, zooko zo...@zooko.com wrote: I'd like to push back on the other risk, that someone might figure out how to make MD5 second-pre-images. I don't think this is a risk that we need to urgently address, and I've written a short note explaining why. This note is incomplete, badly edited, has not been peer-reviewed, and is not ready for publication, but I thought it might help folks evaluate how urgent it is to upgrade from MD5, so here it is. I don't think it's urgent to fix it, but I think it's a good security hardening effort with very little downside and very little chance of regression. However, as I said if Holger, or anyone else, has a concern about the affects of adding this bit of security hardening to give us a safety net again then I simply won't do it in the simple API. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] a plea for backward-compatibility / smooth transitions
- - Distribute / setuptools merge, e.g. cratering folks who use a distro-managed 'python-distribute' package. This is the biggest issue. I wasn't involved and it could have been handled better sure. what issue are we talking about exactly? I'm aware of the Import setuptools problem during upgrades from distribute to setuptools: https://github.com/pypa/pip/issues/1064 pip-1.4 does prevent that now (released last week) but you specifically mentioned the linux distro name, python-distribute? I'm guessing maybe you're concerned with the scenario of how upgrading some project (that depends on setuptools/distribute) forces an unintended upgrade of distribute on a system where distribute is system-managed. Is there a link to a tracker issue for that somewhere? 3 solutions to ponder: 1) unreleasing the distribute-0.7.3 wrapper, and making any upgrades from distribute to setuptools manual. 2) changing pip's -U logic to not recursively upgrade satisfied dependencies, but that's a *big* change (it's a long story, but there are tentative plans in 1.5 for changing it) 3) in a patch release, include a special case for distribute to *not* upgrade if satisfied (when distribute is a dependency in the requirement set). Marcus ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] a plea for backward-compatibility / smooth transitions
- - Distribute / setuptools merge, e.g. cratering folks who use a distro-managed 'python-distribute' package. This is the biggest issue. I wasn't involved and it could have been handled better sure. what issue are we talking about exactly? I'm aware of the Import setuptools problem during upgrades from distribute to setuptools: https://github.com/pypa/pip/issues/1064 pip-1.4 does prevent that now (released last week) but you specifically mentioned the linux distro name, python-distribute? I'm guessing maybe you're concerned with the scenario of how upgrading some project (that depends on setuptools/distribute) forces an unintended upgrade of distribute on a system where distribute is system-managed. Is there a link to a tracker issue for that somewhere? but on 2nd thought, this is not a new problem. this would always been happening, just not involving a transition to setuptools. so I imagine, your concern was just the Import error? ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] a plea for backward-compatibility / smooth transitions
On Jul 29, 2013, at 3:14 PM, Donald Stufft don...@stufft.io wrote: On Jul 29, 2013, at 2:57 PM, zooko zo...@zooko.com wrote: I'd like to push back on the other risk, that someone might figure out how to make MD5 second-pre-images. I don't think this is a risk that we need to urgently address, and I've written a short note explaining why. This note is incomplete, badly edited, has not been peer-reviewed, and is not ready for publication, but I thought it might help folks evaluate how urgent it is to upgrade from MD5, so here it is. I don't think it's urgent to fix it, but I think it's a good security hardening effort with very little downside and very little chance of regression. However, as I said if Holger, or anyone else, has a concern about the affects of adding this bit of security hardening to give us a safety net again then I simply won't do it in the simple API. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig Somewhat relevant to the question at hand: http://valerieaurora.org/hash.html (Yes it lists sha-2 as weakened, which it is. However sha-3 isn't widespread enough for us :( ) - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] a plea for backward-compatibility / smooth transitions
On Mon, Jul 29, 2013 at 04:33:11PM -0400, Donald Stufft wrote: Somewhat relevant to the question at hand: http://valerieaurora.org/hash.html Heh heh. That page is cited in my note. My note is kind of a response to that page, showing that the history of pre-image attacks is completely different than the history of collision attacks. (Yes it lists sha-2 as weakened, which it is. However sha-3 isn't widespread enough for us :( ) There's no reason to worry about SHA-2. In my opinion, there's no particular reason to think that it will be made vulnerable to collisions within the next decade! By the way, I'm a co-author of a secure hash function -- BLAKE2: https://blake2.net/ The intent of BLAKE2 is to be as secure as SHA-3 but as fast as MD5. Not only is it as fast as MD5, but it also has an optional parallel mode that can go 4 or 8 times as fast as MD5 by using 4 or 8 CPU cores! It is currently being adopted for uses like data deduplication, archiving, and distributed filesystems, where the data can be large (terabytes or more), and the performance of the hash function is a bottleneck. I don't think Python packaging has such needs, and BLAKE2 is not a standard like SHA-2 and SHA-3, so I'm not pushing to add support for it. Regards, Zooko ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] a plea for backward-compatibility / smooth transitions
On 30 Jul 2013 05:15, Donald Stufft don...@stufft.io wrote: On Jul 29, 2013, at 2:57 PM, zooko zo...@zooko.com wrote: I'd like to push back on the other risk, that someone might figure out how to make MD5 second-pre-images. I don't think this is a risk that we need to urgently address, and I've written a short note explaining why. This note is incomplete, badly edited, has not been peer-reviewed, and is not ready for publication, but I thought it might help folks evaluate how urgent it is to upgrade from MD5, so here it is. I don't think it's urgent to fix it, but I think it's a good security hardening effort with very little downside and very little chance of regression. However, as I said if Holger, or anyone else, has a concern about the affects of adding this bit of security hardening to give us a safety net again then I simply won't do it in the simple API. I'm thinking that may be the way to go - treat verified SSL as our final stop-gap for the simple API and focus on hardening the next generation APIs. This is more for social reasons than strictly technical ones. I think you're right this particular change is unlikely to break anything, but there are also enough genuinely essential changes needed that we should avoid unnecessary flux in other areas. In this case, I think the need for a pre-image attack that still produces a working download and an old installer that isn't using verified SSL but can check SHA256 hashes reduces the attack window to a point where I'm prepared to live with the use of MD5 as a known risk. Cheers, Nick. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] a plea for backward-compatibility / smooth transitions
Paul Moore p.f.moore at gmail.com writes: Personally, none of the changes have detrimentally affected me, so my opinion is largely theoretical. But even I am getting a little frustrated by the constant claims that what we have now is insecure and broken, and must be fixed ASAP. FWIW, +1. You may be paranoid, but not everyone has to be (or suffer the consequences of it). Security issues should be fixed without breaking things in a hassle (which is the policy we followed e.g. for the ssl module, or hash randomization). The whole python.org infrastructure is built on an OS kernel written by someone who thinks security issues are normal bugs. AFAIK there is no plan to switch to OpenBSD. Regards Antoine. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
