Re: [Python-Dev] Impact of Windows PowerShell OneGet ?
On Oct 29, 2014, at 3:34 PM, Glenn Linderman v+pyt...@g.nevcal.com wrote: New package manager from M$... article here http://www.neowin.net/news/windows-10-oneget-a-linux-style-package-management-framework. It seems doubtful that M$ will eliminate .msi (their obscure, hard to configure and use, installation format), so it seems doubtful that the addition of OneGet will _force_ any changes to Python packaging. However, it does open the question in my mind about whether there will be any _benefits_ of OneGet that would inspire helpful, useful changes to Python packaging. They speak of trusted repositories, and the like, and it sounds like a the various *nix package managers (apt-get, et alia), but perhaps allowing multiple repositories rather than just a single source vendor repository (I'm actually not sure if *nix package managers allow multiple repositories or not, but from the way people talk about them, it always sounds like a distribution also provides a repository of additional packages). trusted repositories sounds more like Perl's CPAN. One of the links http://blogs.technet.com/b/windowsserver/archive/2014/04/03/windows-management-framework-v5-preview.aspx contains this quote: This first version of OneGet installs and searches software from Chocolatey repositories. Support of additional repositories will come in subsequent versions. I have no clue what a Chocolatey repository is (yet, will Google), but unknown others will come, it says... whether it is possible to write a repository plugin such that Perl's CPAN or Python's PyPI or other preexisting repositories can be accessed is not clear. The relationship between PowerShell and OneGet is not clear either... is OneGet written in PowerShell, or is PowerShell just one way to invoke OneGet, or??? Just a heads up. It appears to be a package manager manager. Chocolatey is one of the third party package managers available on Windows. I also just learned that OneGet is apparently OSS and developed on github (https://github.com/OneGet/oneget https://github.com/OneGet/oneget). --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of C compilers for Python on Windows
This sounds like something good for packaging.python.org On Oct 29, 2014, at 4:05 PM, Paul Moore p.f.mo...@gmail.com wrote: On 29 October 2014 15:31, Nathaniel Smith n...@pobox.com wrote: You can use Express editions of Visual Studio. IIUC, the express edition compilers are 32-bit only, and what you actually want are the SDK compilers: https://github.com/cython/cython/wiki/64BitCythonExtensionsOnWindows These are freely downloadable by anyone, no msdn subscription required, but only if you know where to find them! AFAICT the main obstacle to using MSVC to build python extensions (assuming it can handle your code at all) is not anything technical, but rather that there's no clear and correct tutorial on how to do it, and lots of confusion and misinformation circulating. Would it help if I wrote a document explaining how to set up the MS compilers (free and paid for) to allow building of Python extensions? There are a few provisos: 1. A lot of it will be pretty trivial: If you have Vistal Studio (full edition), install it. Done. 2. It will be out of date very fast as the situation for Python 3.5+ will be trivial across the board. 3. I don't have anywhere particularly authoritative to host it (maybe the Python Wiki?) and it could easily get lost in the huge swamp of variously outdated, over-complicated, or otherwise alternative documents available. Ideally I'd like someone to suggest an official location I could use. I don't want to do this if it won't be useful, as it'll take me a bit of effort to confirm the process for the only non-trivial scenario (64-bit Python 3.3/3.4 with free tools). But if people think it would help, that's fine, I volunteer. Paul PS Even if I don't get positive feedback, I may just say to heck with it and do it anyway, because it *is* so trivial :-) I just won't promise. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of C compilers for Python on Windows
On Oct 29, 2014, at 6:09 PM, Paul Moore p.f.mo...@gmail.com wrote: On 29 October 2014 20:26, Donald Stufft don...@stufft.io wrote: This sounds like something good for packaging.python.org Yeah, I wondered about that. I'll work up a patch for that. But the more I think about it, it really is trivial: - For non-free MSVC, install the appropriate version, and everything just works. - For Python 2.7 (32 or 64 bit), install the compiler for Python 2.7 package and everything just works as long as you're using setuptools. - For 32 bit Python 3.2-3.4, install Visual Studio Express and everything just works. - For 64 bit Python 3.2-3.4, install the SDK, set some environment variables, and everything just works. - For Python 3.5+, install the current Visual Studion Express and everything just works. (I propose to deem Python 2.7 without setuptools as unsupported) The only things I can see that need expansion are: 1. The precise versions to use (trivial to add, I'm just to lazy to check right now). 2. Where to get VS 2010 Express (as it's no longer supported or available from MS). 3. How to set up the SDK environment for 64-bit Python 3.2-3.4. Before I do a write-up, I want to set up clean VMs with these configurations, so that I can confirm the details. But I don't expect any problems, as I've done them all before. Paul. I think it’d be good even if considered trivial. I know I certainly have no idea which pieces I needed to do that. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of C compilers for Python on Windows
On Oct 29, 2014, at 7:02 PM, Ethan Furman et...@stoneleaf.us wrote: On 10/29/2014 03:46 PM, Paul Moore wrote: On 29 October 2014 22:19, Ethan Furman et...@stoneleaf.us wrote: - where one should be at when one starts the compile process I don't understand this. It's just pip wheel foo to build a wheel for foo (which will be downloaded), or pip wheel . or python setup.py bdist_wheel as you prefer for a local package. Hmmm... That looks like it's for installing/compiling somebody else's package. Is that last command sufficient to prepare one's own wheel for uploading to PyPI, or there something else to do? Generally for uploading to PyPI you do ``python setup.py bdist_wheel``, though I don’t think there’d be any bad thing if you used pip wheel. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XP buildbot problem cloning from hg.python.org
I have an idea, can you run https://bpaste.net/show/c5d7cd102f5b and tell me what it outputs? Both on a machine that works and one that doesn’t. On Oct 25, 2014, at 2:14 AM, David Bolen db3l@gmail.com wrote: Donald Stufft don...@stufft.io writes: What version of OpenSSL is it using. I'm using the pre-built Windows Mercurial installer, but if I unpack the included library.zip, the SSLEAY32.DLL shows version 0.9.8r. This is from the 3.1.2 install I just did a few hours ago. It appears that hg 2.5.2 on my other XP box also has 0.9.8r. The prior buildbot version (1.6.2) looks like it had 0.9.8o. I also got around to trying a manual clone on the Windows 7 buildbot, and it worked fine, even with the older hg 1.6.2. So it seems to correlate with XP more than anything else at the moment. -- David ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XP buildbot problem cloning from hg.python.org
Is this using HTTPS or SSH. On Oct 24, 2014, at 11:47 PM, David Bolen db3l@gmail.com wrote: Starting yesterday, my XP buildbot began failing to execute clone operations against hg.python.org. There's not a lot of data being given aside from a transaction abort message (and my buildbot log showing the hg command exiting), and I'm wondering if something may be amiss on the server or its configuration? Note that this is a full clone (which for some reason the Windows buildbots seem to fall back on with some frequency) and can take quite a while. My Windows 7 buildbot is ok so far but it's still doing incremental pulls over the same time period. I've got two separate Internet connections here and have tried routing over both so I don't think it's a network issue. I've completely flushed the local build trees and rebooted the buildbot. Is there anything that might be available on the server to see if there are errors being logged? Or anything that could have changed configuration wise recently (maybe timeout related or something)? I'm running a bit low of items to try to change or reset on the buildbot side. Thanks. -- David ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XP buildbot problem cloning from hg.python.org
What version of OpenSSL is it using. On Oct 25, 2014, at 1:00 AM, David Bolen db3l@gmail.com wrote: David Bolen db3l@gmail.com writes: which appears to die mid-stream while receiving the manifests. So I'm sort of hoping there might be some record server-side as to why things are falling apart mid-way. Just to follow-up to myself, I get the same same error trying to do a clone from my own personal XP machine rather than the buildbot (which is a VM). I've had the issue with hg 1.6.2, 2.5.2 and 3.1.2. However, the same clones completely successfully under OSX and Linux. So that's sort of strange. -- David ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] https://docs.python.org/3/using/index.html not linking correctly
I'm on my phone but docs is served via fastly. Issues could be POP specific. On Oct 20, 2014, at 7:38 PM, Oleg Broytman p...@phdru.name wrote: On Tue, Oct 21, 2014 at 12:29:45AM +0100, MRAB pyt...@mrabarnett.plus.com wrote: On 2014-10-21 00:09, Eli Bendersky wrote: On Mon, Oct 20, 2014 at 1:01 PM, Terry Reedy tjre...@udel.edu mailto:tjre...@udel.edu wrote: If I go to https://docs.python.org/3/using/index.html and click on any of the TOC entries, I get 'connecting' indefinitely. This problem seems unique to this file. I tried several other index files and clicking am entry brings up the corresponding page almost immediately. Works fine for me, Terry, in both Chrome and Firefox. Could be something in your browser/caching? Try private mode or whatever that's called in your browser of choice. Just tried the first link: https://docs.python.org/3/using/cmdline.html I also get 'Connecting' indefinitely. So I tested the link here: http://www.downforeveryoneorjustme.com/ It said: It's not just you! http://https looks down from here. I think this is a limitation of isup.me: it doesn't like https:// URLs -- either use http:// or no protocol prefix at all. Oleg. -- Oleg Broytmanhttp://phdru.name/p...@phdru.name Programmers don't die, they just GOSUB without RETURN. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Disabling SSL 3.0
A big security breach of SSL 3.0 just dropped a little while ago (named POODLE). With this there is now no ability to securely connect via SSL 3.0. I believe that we should disable SSL 3.0 in Python similarly to how SSL 2.0 is disabled, where it is disabled by default unless the user has explicitly re-enabled it. The new attack essentially allows reading the sensitive data from within a SSL 3.0 connection stream. It takes roughly 256 requests to break a single byte so the attack is very practical. You can read more about the attack here at the google announcement [1] or the whitepaper [2]. [1] http://googleonlinesecurity.blogspot.com/2014/10/this-poodle-bites-exploiting-ssl-30.html [2] https://www.openssl.org/~bodo/ssl-poodle.pdf --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of C compilers for Python on Windows
On Oct 10, 2014, at 6:59 PM, Steve Dower steve.do...@microsoft.com wrote: Cross compilation is a valid issue, but I hope that build services like Appveyor also help out here. There is regular talk about the PSF/PyPI providing something similar, though I have doubts about its feasibility under any model other than renting a preconfigured VM. I don't see any reason why projects couldn't apply to the PSF for a grant to cover Appveyor costs or the expenses of similar services. I know very little about Windows, but we can spin up arbitrary Windows boxes, build something on them, and then shut them down as long as there’s some way to automate logging into a Windows box and running some commands… Normally I’d do this with SSH but I don’t know if Windows has anything like that. IOW we can totally spin up preconfigured VMs for a Windows build service. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] shebang policy, and pip
On Oct 8, 2014, at 6:16 AM, Christian Tismer tis...@stackless.com wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Howdy, this question is a bit about general policy which is not yet covered in the python recommendations: I see projects which do check-ins like get rid of shebang lines and they remove those lines from non-script sources. It is not always clear to me what to do, so I tend to leave those lines in per default, in order not to waste time thinking about it, but well, today I was confronted with that. Digging a bit deeper shows the following: python docs: No mention of shebang, but for Windows. https://docs.python.org/3/search.html?q=shebangcheck_keywords=yesarea=default https://docs.python.org/3/using/windows.html?highlight=shebang Google's python style guide also says when a shebang is needed, but does not forbid it. Pep 394 explains how to use shebang, but still nothing about not using it. http://legacy.python.org/dev/peps/pep-0394/ So is there anything officially preferred, and should that go into pep 8? Some editors can use shebang lines to control syntax highlighting or linting (mine for example will lint different for python2 vs python3 shebangs). Special case with pip - -- I was looking through my installed packages and wondered quite much about pip: Pip has a shebang in the __init__ file, but no shebang in the __main__ file. I guess this is wrong and should be in the executable file, which is __main__ . *puts on pip developer hat* I’m guessing that comes from when pip used to be a single file and from before we dropped support for Pythons that didn’t support python -m. It’s probably nonsense cruft that’s just been left over and hadn’t been managed to be deleted now. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] shebang policy, and pip
On Oct 8, 2014, at 3:36 PM, Wes Turner wes.tur...@gmail.com wrote: On Oct 8, 2014 7:20 AM, Donald Stufft don...@stufft.io mailto:don...@stufft.io wrote: On Oct 8, 2014, at 6:16 AM, Christian Tismer tis...@stackless.com mailto:tis...@stackless.com wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Howdy, this question is a bit about general policy which is not yet covered in the python recommendations: I see projects which do check-ins like get rid of shebang lines and they remove those lines from non-script sources. It is not always clear to me what to do, so I tend to leave those lines in per default, in order not to waste time thinking about it, but well, today I was confronted with that. Digging a bit deeper shows the following: python docs: No mention of shebang, but for Windows. https://docs.python.org/3/search.html?q=shebangcheck_keywords=yesarea=default https://docs.python.org/3/search.html?q=shebangcheck_keywords=yesarea=default https://docs.python.org/3/using/windows.html?highlight=shebang https://docs.python.org/3/using/windows.html?highlight=shebang Google's python style guide also says when a shebang is needed, but does not forbid it. Pep 394 explains how to use shebang, but still nothing about not using it. http://legacy.python.org/dev/peps/pep-0394/ http://legacy.python.org/dev/peps/pep-0394/ So is there anything officially preferred, and should that go into pep 8? Some editors can use shebang lines to control syntax highlighting or linting (mine for example will lint different for python2 vs python3 shebangs). Does it support shebang lines like: #!/usr/bin/env python ? Sure. Though it doesn’t resolve it to determine what that would actually execute, it just uses the name of the python in the shebang as heuristics. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Backporting ensurepip to 2.7, Which commands to install?
I'm working on the backport of ensurepip to Python 2.7, and I realized that I'm not sure which commands to install. Right now by default pip (outside of the context of ensurepip) will install pip, pip2, and pip2.7 if installed in Python 2.7. In Python 3's ensurepip we modified it so that it would install pip3, and pip3.4, but *not* pip if it was an install, and only pip3.4 if it was an alt install. My question is, does this behavior make sense for ensurepip in 2.7? Or should it also install the pip command if it is an install? --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backporting ensurepip to 2.7, Which commands to install?
Ok, so neither Python 2.7 nor Python 3.x’s ensure pip command will install a ``pip`` binary by default without a flag. That's fine with me, just wanted to make sure it made sense for Python 2.x. Thanks! On Oct 3, 2014, at 8:31 PM, Guido van Rossum gu...@python.org wrote: That is copying the (alt)install targets of Python's own Makefile, and I think those are exactly right. On Oct 3, 2014 3:07 PM, Donald Stufft don...@stufft.io mailto:don...@stufft.io wrote: I'm working on the backport of ensurepip to Python 2.7, and I realized that I'm not sure which commands to install. Right now by default pip (outside of the context of ensurepip) will install pip, pip2, and pip2.7 if installed in Python 2.7. In Python 3's ensurepip we modified it so that it would install pip3, and pip3.4, but *not* pip if it was an install, and only pip3.4 if it was an alt install. My question is, does this behavior make sense for ensurepip in 2.7? Or should it also install the pip command if it is an install? --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org mailto:Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org https://mail.python.org/mailman/options/python-dev/guido%40python.org --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backporting ensurepip to 2.7, Which commands to install?
Whoops, I misred. So to be clear, you think: install - pip, pip2, pip2.7 altinstall - pip2.7 On Oct 3, 2014, at 8:46 PM, Guido van Rossum gu...@python.org wrote: That's not what I meant. Python 2.7 does install python unless you use altinstall. On Oct 3, 2014 5:33 PM, Donald Stufft don...@stufft.io mailto:don...@stufft.io wrote: Ok, so neither Python 2.7 nor Python 3.x’s ensure pip command will install a ``pip`` binary by default without a flag. That's fine with me, just wanted to make sure it made sense for Python 2.x. Thanks! On Oct 3, 2014, at 8:31 PM, Guido van Rossum gu...@python.org mailto:gu...@python.org wrote: That is copying the (alt)install targets of Python's own Makefile, and I think those are exactly right. On Oct 3, 2014 3:07 PM, Donald Stufft don...@stufft.io mailto:don...@stufft.io wrote: I'm working on the backport of ensurepip to Python 2.7, and I realized that I'm not sure which commands to install. Right now by default pip (outside of the context of ensurepip) will install pip, pip2, and pip2.7 if installed in Python 2.7. In Python 3's ensurepip we modified it so that it would install pip3, and pip3.4, but *not* pip if it was an install, and only pip3.4 if it was an alt install. My question is, does this behavior make sense for ensurepip in 2.7? Or should it also install the pip command if it is an install? --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org mailto:Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org https://mail.python.org/mailman/options/python-dev/guido%40python.org --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 3.5 release schedule PEP
On Sep 26, 2014, at 3:09 AM, Paul Moore p.f.mo...@gmail.com wrote: On 26 September 2014 01:38, Donald Stufft don...@stufft.io wrote: Either way I'm fairly commited to making --user the default, the only question on my mind is what exactly does that look like (e.g. does root get --user by default?) and how we get from where we are now to that point. I think that raising an error here is fairly unfriendly when we can know with pretty good certainity what the user wanted (and they can explicitly declare that if they want too). A couple of points come to mind (although they may be more suitable for distutils-sig). 1. Do user installs leak into virtualenvs? If so, then in effect --use-system-packages is switched back on again if --user installs become the norm. Which is almost certainly not what is wanted. 2. pip install should default to not being --user when run from within a virtualenv (same logic as the isolated Python case, but much more important that behaviour remains as now, because the whole *point* of virtualenvs is to isolate). Note that both of these points apply both to venv and virtualenvs. 1. No they don’t leak as far as I’m aware. 2. Yea, I think we throw an error when you use —user inside a virtual environment. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 3.5 release schedule PEP
On Sep 26, 2014, at 9:53 AM, Paul Moore p.f.mo...@gmail.com wrote: On 26 September 2014 14:31, Donald Stufft don...@stufft.io wrote: Yea, I think we throw an error when you use —user inside a virtual environment. So if --user became the default, what would happen? I'd like pip inside a virtualenv to install into the environment without needing a --system flag. Is that the intention? Paul default = “—user” if not running_inside_virtualenv() else “—system --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Microsoft Visual C++ Compiler for Python 2.7
Awesome! On Sep 26, 2014, at 2:01 PM, Steve Dower steve.do...@microsoft.com wrote: Hi all, (This is advance notice since people on this list will be interested. Official announcements are coming when setuptools makes their next release.) Microsoft has released a compiler package targeting Python 2.7 (i.e. VC9). We've produced this package to help library developers build wheels for Windows, but also to help users unblock themselves when they need to build C extensions themselves. The Microsoft Visual C++ Compiler for Python 2.7 is available from: http://aka.ms/vcpython27 This package contains all the tools and headers required to build C extension modules for Python 2.7 32-bit and 64-bit (note that some extension modules require 3rd party dependencies such as OpenSSL or libxml2 that are not included). Other versions of Python built with Visual C++ 2008 are also supported. You can install the package without requiring administrative privileges and, with the latest version of setuptools (when it releases), use tools such as pip, wheel, or a setup.py file to produce binaries on Windows. Unfortunately, this package isn't necessarily going to help with building CPython 2.7 itself, as the build process is complicated and Visual Studio 2008 is practically required. However, as most people aren't building CPython on Windows, this isn't a huge issue. This compiler package should be sufficient for most extension modules. I should also point out that VC9 is no longer supported by Microsoft. This means there won't be any improvements or bug fixes coming, and there's no official support offered. Feel free to contact me directly steve.do...@microsoft.com if there are issues with the package. Cheers, Steve ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 3.5 release schedule PEP
On Sep 25, 2014, at 4:54 AM, Antoine Pitrou solip...@pitrou.net wrote: On Thu, 25 Sep 2014 07:34:31 +0100 Paul Moore p.f.mo...@gmail.com wrote: On 25 September 2014 02:08, Antoine Pitrou solip...@pitrou.net wrote: Indeed. Moving towards having --user as the norm is definitely something we want to look at for pip. One of the biggest concerns is how well-exercised the whole user site directory area is in practice. What do you mean by well-exercised? Basically, although --user is available in pip (and the underlying facilities in Python have been around for some time), it's difficult to gauge how many people are using them, and as a result what level of testing has happened in real-life situations. I'm using it often. I'm also unsure how broken it could be. The user site-packages is just another site-packages directory. Broken like the prefix problem :) Basically people have Python in a ton of different configurations and it’s hard to figure out if —user will work out of the box in all of them or not. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 3.5 release schedule PEP
On Sep 25, 2014, at 11:54 AM, Paul Moore p.f.mo...@gmail.com wrote: On 25 September 2014 16:43, Donald Stufft don...@stufft.io wrote: Basically people have Python in a ton of different configurations and it’s hard to figure out if —user will work out of the box in all of them or not. I guess that Using the python.org Python installer on Windows is a limited enough subset that we probably could check that --user worked in that situation. The problem is, how do we implement it? A special case so that pip defaults to --user sometimes, but not others? (I'm strongly against that) Leave the default as not --user and document that Windows users with Python in Program Files should always specify --user? (I'm against that because it makes the documentation highly confusing, and we've just done a lot of work to simplify it). Basically, I'd like to hold off moving to Program Files as a default until *after* we have enough confidence in user installs that we are willing to switch pip to --user as the default behaviour everywhere. And yes, I'm aware that the first we in that was python-dev and the second was PyPA. And that expecting python-dev to wait for PyPA to approve the change may well be unacceptable. Paul My thoughts on the pip side has basically always been that pip should either: 1) Just always default to —user and add a —system or similar flag, this is super easy to change but is a backwards incompatible change and would need to go through a deprecation window. 2) Switch to —user based on if the user has permission to write to the site-packages or not. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 3.5 release schedule PEP
On Sep 25, 2014, at 6:44 PM, Chris Barker chris.bar...@noaa.gov wrote: On Thu, Sep 25, 2014 at 9:00 AM, Donald Stufft don...@stufft.io mailto:don...@stufft.io wrote: 1) Just always default to —user and add a —system or similar flag, this is super easy to change but is a backwards incompatible change and would need to go through a deprecation window. Maybe would have been the way to go to begin with, but I think backwards compatibility should probably trump better -- even with a deprecation window. 2) Switch to —user based on if the user has permission to write to the site-packages or not. ouch -- no. Why not a clear error message if pip can't write to site-packages -- something like: pip doesn't have permissions to write to the system location. If you want to install this package system-wide, you need to run pip with admin priviledges (and example here if it's easy), if you want to install for this user only, pass the --user flag to pip install In the face of ambiguity, refuse the temptation to guess.” I fairly strongly believe that the current default is doing a great disservice to users. I believe that for *most* people --user is the correct option for them to be using and the fact that it's not the default and requires opt in is a historical artifact more than anything else. Immediately switching to --user as a default (outside of a virtual environment) would break two important use cases: - A root/admin user attempting to install into a global site-packages - A user who uses complete Python installations to isolate themselves such as those created by pyenv. It would techincally break anyone relying on the fact that pip will currently throw an error if you attempt to ``pip install`` something without sudo, however I do not consider that to be an actual use case that is going to have wide signifcance. So we get down to how do we not break the two important use cases above. At first I thought about detecting uid 0 (and something similar on Windows?) and turning off the --user default in those cases. However that still left us dealing with something for the second use case. After thinking about it I realized there isn't a good way to determine if something is a user controlled Python installation instead of a system controlled one. This lead me to the realization that something that could be used to detect this is whether or not the current user has the ability to write to the site-packages as a means of determining is my current user allowed to manage that particular Python. Either way I'm fairly commited to making --user the default, the only question on my mind is what exactly does that look like (e.g. does root get --user by default?) and how we get from where we are now to that point. I think that raising an error here is fairly unfriendly when we can know with pretty good certainity what the user wanted (and they can explicitly declare that if they want too). However this particular thing is somewhat offtopic for this list and the particulars of pip moving to --user by default is best discussed on our issue tracker here: https://github.com/pypa/pip/issues/1668. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 3.5 release schedule PEP
On Sep 24, 2014, at 4:24 PM, Steve Dower steve.do...@microsoft.com wrote: Paul Moore wrote: On 24 September 2014 14:16, Mike Miller python-...@mgmiller.net wrote: It has been a supported option for just shy of 15 years on 2.X... most if not all the bugs (setuptools) were fixed a decade ago, and right now thousands, if not millions of people are running it under Program Files right now. I can vouch for several thousand because a company I work for distributes Python and pip there for its customers all around the world w/o issue. One thing that I presume would be an issue. Isn't Program Files protected in newer versions of Windows? I haven't tested this myself, so I may be wrong about this. So take the following with a pinch of salt. It's protected very well in newer versions. You typically need to be an administrator AND have opted in to being able to modify system files without warning. Assuming so, that means that if Python is installed there, the standard pip install XXX would not work unless run in an elevated shell. We are currently trying to focus on a unified message for how users should install distributions from PyPI, by using pip install. I'm not sure it's a good idea to complicate that message yet by adding provisos about managing the system Python (which is the only one most Windows users will use). This is my main concern. Until pip install --user is the default (or the fallback if there are no write permissions on the destination), a default that locks users out of the simplest PyPI experience is a genuine problem. Yes, users can elevate to run pip, but I'd prefer pip to use elevation if it has it and to use per-user if not. There also isn't a great story for per-user Python installs on Windows, but that becomes fairly cheap with the installer rewrite. I know this is only the same situation as Unix users have, but Windows users don't have a distro offering packaged versions of PyPI modules. I also know we should be moving towards promoting --user, but I don't think we're quite ready for that yet. And my speculation doesn't compete with your real-life experience, certainly. But I would suggest carefully checking before making the switch. A good reason to decide early on a change like this, or at least to promote it as an option in 3.5 and make it the default in 3.6. One thing about *nix is even though you can’t write to your normal Python install location without root, invoking pip with permissions (assuming you have them) is as easy as prefacing it with ``sudo`` in most cases. Does Windows have an equivalent or do you need to launch a whole new shell? --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 3.5 release schedule PEP
On Sep 23, 2014, at 9:48 PM, Nick Coghlan ncogh...@gmail.com wrote: On 24 September 2014 03:05, Steve Dower steve.do...@microsoft.com mailto:steve.do...@microsoft.com wrote: Larry Hastings wrote: On 09/19/2014 03:31 PM, Barry Warsaw wrote: I think we need a Python 3.5 Release Schedule PEP. Just checked it in as PEP 478. It should show up here in a few minutes: http://legacy.python.org/dev/peps/pep-0478/ Key facts: . Beta 1 is May 24th 2015, about a month after the end of the PyCon US 2015 sprints. . Final release is September 13, 2015, just over a year from now. Comments? Martin is no longer producing the Windows installers - that task has been handed to me. I'm planning to have a rewritten installer (also in the same repo) that should be easier to modify and maintain, as well as being able to produce alternative packages (such as a Python 3.5 or stdlib merge module, for example), though that doesn't necessarily need to go into the PEP. I'm also considering/experimenting with installing into Program Files by default, but I suspect that isn't going to work out yet. I'd like to move the Windows versions onto the next release of VC (currently VC 14 until the branding team figures out what to call it). There isn't a promised RTM date for VC 14 yet, so it looks like the best available compiler by Beta 1 will be a Go Live RC. (The Go Live marking basically means we think this is ready for use, but expect a round of minor updates/fixes soon - the compiler is least likely to be updated, my guess is that it'll be Visual Studio UI mostly). I personally don't have any qualms about using the RC compiler for Beta 1, and Beta 2 will almost certainly use VC 14 RTM, but I know when I proposed this topic that some people were concerned about having the final version available for Python 3.5 Beta. So far I've been building regularly with internal versions of VC and haven't been hitting any major issues with Python (OpenSSL has some issues, but I've been filing bugs on both sides so they should be worked out soon enough). My work is at http://hg.python.org/sandbox/steve.dower (branch: VC14) for anyone interested. For the alphas, I'm contemplating producing two builds (VC 10 and VC 14), but I obviously want to settle on one or the other by Beta. Last time we discussed it, there was strong support for changing compiler, but I have a better idea of the timeline now and it's tighter than I thought... Thoughts, anyone? It's ultimately up to Larry as RM, but I'd personally favour targeting the newer compiler and runtime, even with the slight risk of potentially needing to slip our schedule. There's also a fair amount of wiggle room between the first beta and the first release candidate. Regards, Nick. -- Nick Coghlan | ncogh...@gmail.com mailto:ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org mailto:Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io https://mail.python.org/mailman/options/python-dev/donald%40stufft.io This new compiler has the incredibly awesome feature of being forwards compatible right? Like in 10 years stuff compiled with a newer compiler will still work? --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 3.5 release schedule PEP
On Sep 23, 2014, at 10:14 PM, Steve Dower steve.do...@microsoft.com wrote: This new compiler has the incredibly awesome feature of being forwards compatible right? Like in 10 years stuff compiled with a newer compiler will still work? That's the promise at least :) All the macros that leaked implementation details (like file descriptors) are now isolated so if they change it won't break existing applications. It'll still be possible for newer compiler versions to break them, but the design now obviously discourages it. There's also an official guarantee, so if it is broken in future it'll be treated as a bug. As much as I'd love to make solid promises, I can only pass on the promises that were made to me. But yes, we should have forward compatibility with later MSVC versions, which will help avoid the situation where it's hard to get hold of the right compiler... Top-posted from my Windows Phone Yea I think it makes incredible sense to aim for it then, even if it makes things slip. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 3.5 release schedule PEP
On Sep 22, 2014, at 8:20 PM, Larry Hastings la...@hastings.org wrote: On 09/19/2014 03:31 PM, Barry Warsaw wrote: I think we need a Python 3.5 Release Schedule PEP. Just checked it in as PEP 478. It should show up here in a few minutes: http://legacy.python.org/dev/peps/pep-0478/ http://legacy.python.org/dev/peps/pep-0478/ Key facts: Beta 1 is May 24th 2015, about a month after the end of the PyCon US 2015 sprints. Final release is September 13, 2015, just over a year from now. Comments? It says 3.4.0 all through it. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 394 - Clarification of what python command should invoke
On Sep 19, 2014, at 3:31 AM, Bohuslav Kabrda bkab...@redhat.com wrote: Hi, as Fedora is getting closer to having python3 as a default, I'm being more and more asked by Fedora users/contributors what'll /usr/bin/python invoke when we achieve this (Fedora 22 hopefully). So I was rereading PEP 394 and I think I need a small clarification regarding two points in the PEP: - for the time being, all distributions should ensure that python refers to the same target as python2. - Similarly, the more general python command should be installed whenever any version of Python is installed and should invoke the same version of Python as either python2 or python3. The important word in the second point is, I think, *whenever*. Trying to apply these two points to Fedora 22 situation, I can think of several approaches: - /usr/bin/python will always point to python3 (seems to go against the first mentioned PEP recommendation) - /usr/bin/python will always point to python2 (seems to go against the second mentioned PEP recommendation, there is no /usr/bin/python if python2 is not installed) - /usr/bin/python will point to python3 if python2 is not installed, else it will point to python2 (inconsistent; also the user doesn't know he's running and what libraries he'll be able to import - the system can have different sets of python2-* and python3-* extension modules installed) - there will be no /usr/bin/python (goes against PEP and seems just wrong) I'd really appreciate upstream guidance and perhaps a PEP clarification for distributions that ship both python2 and python3, but can live without python2 (and are not Arch :)). Thanks a lot! I don’t know for a fact, but I assume that as long as Python 2.x is installed by default than ``python`` should point to ``python2``. If Python 3.x is the default version and Python 2.x is the “optional” version than I think personally it makes sense to switch eventually. Maybe not immediately to give people time to update though? --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 394 - Clarification of what python command should invoke
On Sep 19, 2014, at 10:16 AM, Barry Warsaw ba...@python.org wrote: If the user wants to invoke Python 3, it's not hard to type 'python3' and I think that's the message we should be spreading. That already seems pretty ingrained in user habits afaict. My biggest problem with ``python3``, is what happens after 3.9. I know Guido doesn’t particularly like two digit version numbers and it’s been suggested on this list that instead of 3.10 we’re likely to move directly into 4.0 regardless of if it’s a “big” change or not. If that is the case, then all of the user education, ui, etc around ``python3`` would then need to be again updated to ``python4`` *OR* we’d need a ``python3`` bin which points to ``python4``. If there’s a call to action for at some point moving ``python`` to invoke Python 3.x at some point then hopefully at that point Python 4.x could just be ``python``. All of this assuming of course that 4.0 isn’t a major break like 3.0 and that we do 4.0 instead of 3.10 as has been suggested. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 394 - Clarification of what python command should invoke
On Sep 19, 2014, at 8:02 PM, Greg Ewing greg.ew...@canterbury.ac.nz wrote: Donald Stufft wrote: My biggest problem with ``python3``, is what happens after 3.9. Python2 technically includes 1.x versions as well, so it wouldn't be unprecedented for python3 to imply versions beyond 3.x. It would be a bit confusing, though. My problem isn’t that it *includes* it, it’s that either people have to go through the mess to update all of their things to ``python4`` at some point in the future, or Python 3.x will need to eventually mean ``python``. Well that or Python 4.x has a ``python3`` binary. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Proposed schedule for 3.4.2
On Sep 8, 2014, at 6:20 PM, Nick Coghlan ncogh...@gmail.com wrote:
On 9 Sep 2014 04:00, Barry Warsaw ba...@python.org
mailto:ba...@python.org wrote:
This would need to be updated first, once it *did* take such an argument,
this would be accomplished by:
context = ssl.create_default_context()
context.verify_mode = CERT_OPTIONACERT_NONE
context.verify_hostname = False
urllib.request.urlopen(https://something-i-apparently-dont-care-much-about
https://something-i-apparently-dont-care-much-about/,
context=context)
There's probably an ugly hack possibility that uses unittest.mock.patch. ;)
We could actually make it an official hack:
import urllib.request
urllib.request.urlopen = urllib.request._unverified_urlopen
Or else the user can just change the code to call the unverified one directly.
All we'd have to do is keep the existing version that doesn't validate certs
properly around under the name _unverified_urlopen.
I like this for a few reasons:
1. It doesn't get much easier than calling function A instead of function B
2. Monkeypatching lets you do a process global hack
3. The name tells you exactly why this is a bad idea
4. It's easy to grep for later after you fix your certs
5. The leading underscore acts as a strong keep away signal
6. The leading underscore makes it clear this function may not always be
available (e.g. Jython, older versions of Python)
If someone wants to do this, can’t they write their own 6 line function?
import ssl
import urllib.request
_real_urlopen = urllib.request.urlopen
def _unverified(*args, **kwargs):
if not kwargs.keys() {“context”, “cafile”, “capath”, “cadefault”}:
ctx = ssl.create_default_context()
ctx.verify_mode = CERT_NONE
ctx.verify_hostname = False
kwargs[“context”] = ctx
return _real_urlopen(*args, **kwargs)
---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Proposed schedule for 3.4.2
On Sep 8, 2014, at 6:43 PM, Nick Coghlan ncogh...@gmail.com wrote:
On 9 Sep 2014 08:30, Donald Stufft don...@stufft.io
mailto:don...@stufft.io wrote:
If someone wants to do this, can’t they write their own 6 line function?
Unfortunately not, as the domain knowledge required to know what those six
lines should look like is significant.
Keeping the old unsafe behaviour around with a more obviously dangerous name
is much simpler than explaining to people Here, copy this chunk of code you
don't understand.
If we were starting with a blank slate there's no way we'd offer such a
thing, but as Jim pointed out, we do want to make it relatively easy for
Standard Operating Environment maintainers to hack around it if necessary.
Cheers,
Nick.
import ssl
import urllib.request
_real_urlopen = urllib.request.urlopen
def _unverified(*args, **kwargs):
if not kwargs.keys() {“context”, “cafile”, “capath”, “cadefault”}:
ctx = ssl.create_default_context()
ctx.verify_mode = CERT_NONE
ctx.verify_hostname = False
kwargs[“context”] = ctx
return _real_urlopen(*args, **kwargs)
---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Why isn’t documentation with appropriate red warnings a suitable place if we
really must have it? That sounds like a much better solution that some weird
function people monkeypatch. It gives them more control over things (maybe they
have a valid certificate chain, but an invalid host name!), it’ll work across
all Python implementations, and most importantly, it gives us a place where
there is some long form location to be like “yea you really probably don’t want
to be doing this” in big red letters.
Overall I’m -1 on either offering the function or documenting it at all, but if
we must do something then I think documentation is more than enough.
---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backwards compatibility after certificate autovalidation
It’s create_default_context() - https://docs.python.org/3.4/library/ssl.html#ssl.create_default_context On Sep 8, 2014, at 11:40 PM, Guido van Rossum gu...@python.org wrote: The multiple threads going on are confusing (or maybe GMail makes them more confusing), but the architecture you are sketching here sounds good. I can't find get_default_context() in the repo, but perhaps I need to refresh, or perhaps you're talking about a design in a PEP. On Mon, Sep 8, 2014 at 8:03 PM, Nick Coghlan ncogh...@gmail.com mailto:ncogh...@gmail.com wrote: On 9 Sep 2014 10:48, Jim J. Jewett jimjjew...@gmail.com mailto:jimjjew...@gmail.com wrote: I assume that adding _unverified_urlopen or urlopen(context=...) do provide incremental improvements compatible with the eventual full opt-in. If so, adding them is probably reasonable, but I think the PEP should explicitly list all such approved half-measures as a guard against API feature creep. From Guido's and your feedback, I think we may need two things to approve this for 3.4.2 (putting 2.7 aside for now): 1. context parameter support in urllib.request (to opt out on a per-call basis) 2. a documented way to restore the old behaviour via sitecustomize (which may involve monkeypatching) The former change seems non-controversial. I think the more fine-grained solution for the latter can wait until 3.5 (and will be an independent PEP), we just need an interim workaround for 3.4 that could conceivably be backported to 2.7. On that front, ssl currently has two context factories: get_default_context() and _get_stdlib_context. One possible option would be to: 1. Rename _get_stdlib_context to _get_unverified_context 2. Add _get_https_context as an alias for get_default_context Opting out on a per-call basis means passing an unverified context. Opting out globally would mean monkeypatching _get_https_context to refer to _get_unverified_context instead. These would both be documented as part of transition, but with clear security warnings. The use of the leading underscores in the names is intended to emphasise you probably don't want to be using this. Regards, Nick. -jJ ___ Python-Dev mailing list Python-Dev@python.org mailto:Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org mailto:Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org https://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (python.org/~guido http://python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On Sep 3, 2014, at 1:54 PM, Guido van Rossum gu...@python.org wrote: On Wed, Sep 3, 2014 at 8:58 AM, R. David Murray rdmur...@bitdance.com mailto:rdmur...@bitdance.com wrote: I'm OK with letting go of this invalid-cert issue myself, given the lack of negative feedback Twisted got. I'll just keep my fingers crossed. I'm with this sentiment (cautiously +1) -- and not just because of Twisted's experience or Glyph's passion. Network security is much more important now than it was five years ago -- and yet Python 2.7 is at least that old. My own experience has changed a lot: five years ago (when I worked at Google!) it was common to find internal services that required SSL but had a misconfigured certificate, and the only way to access those services was to override the browser complaints. Today (working at Dropbox, a much smaller company!) I don't even remember the last time I had to deal with such a browser complaint -- internal services here all redirect to SSL, and not a browser that can find fault with their certs. If I did get a complaint about a certificate I would fire off an email to a sysadmin alerting them to the issue. Let's take the plunge on this issue for the next 2.7 release (3.5 being a done deal). Yes, some people will find that they have an old script accessing an old service which breaks. Surely some of the other changes in the same 2.7 bugfix release will also break some other scripts. People deal with it. Probably 90% of the time it's an annoyance (but no worse than any other minor-release upgrade -- you should test upgrades before committing to them, and if all else fails, roll it back). But at least some of the time it will be a wake-up call and an expired certificate will be replaced, resulting in more security for all. +1, this makes me unreasonably happy. I don't want to start preaching security doom and gloom (the experts are doing enough of that :-), but the scale and sophistication of attacks (whether publicized or not) is constantly increasing, and routine maintenance checks on old software are just one of the small ways that we can help the internet become more secure. (And please let the PSF sysadmin team beef up *.python.org http://python.org/ -- sooner or later some forgotten part of our infrastructure *will* come under attack.) This is an ongoing effort amongst the Infra team, part of the process is moving infrastructure away from hand crafted servers towards servers managed by config management as well as making sure all our services are behind TLS as well. -- --Guido van Rossum (python.org/~guido http://python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On Sep 2, 2014, at 7:47 PM, Glyph Lefkowitz gl...@twistedmatrix.com wrote: On Sep 2, 2014, at 4:28 PM, Nick Coghlan ncogh...@gmail.com mailto:ncogh...@gmail.com wrote: On 3 Sep 2014 09:08, David Reid dr...@dreid.org mailto:dr...@dreid.org wrote: Nick Coghlan ncoghlan at gmail.com http://gmail.com/ writes: Creating *new* incompatibilities between Python 2 Python 3 is a major point of concern. Clearly this change should be backported to Python2. Proposing to break backwards compatibility in a maintenance release (...) As we keep saying, this is not a break in backwards compatibility, it's a bug fix. Yes, systems might break, but that breakage represents an increase in security which may well be operationally important. Not everyone with a working application has the relevant understanding an expertise to know that Python's HTTP client is exposing them to surveillance. These applications should break. That is the very nature of the fix. It is not a compatibility break that the system starts correctly rejecting invalid connections. By way of analogy, here's another kind of breach in security: an arbitrary remote code execution vulnerability in XML-RPC. I think we all agree that any 0day RCE vulnerabilities in Python really ought to be fixed and could be legitimately included without worrying about backwards compatibility breaks. (At least... gosh, I hope so.) Perhaps this arbitrary remote execution looks harmless; the use of an eval() instead of an int() someplace. Perhaps someone discovered that they can do 3 + 4 in their XML-RPC and the server does the computation for them. Great! They start relying on this in their applications to use symbolic values in their requests instead of having explicit enumerations. This can save you quite a bit of code! When the RCE is fixed, this application will break, and that's fine. In fact that's the whole point of issuing the fix, that people will no longer be able to make arbitrary computation requests of your server any more. If that server's maintainer has the relevant context and actually wants the XML-RPC endpoint to enable arbitrary RCE, they can easily modify their application to start doing eval() on the data that they received, just as someone can easily modify their application to intentionally disable all connection security. (Let's stop calling it certificate verification because that sounds like some kind of clerical detail: if you disable certificate verification, TLS connections are unauthenticated and unidentified and therefore insecure.) For what it's worth, on the equivalent Twisted change, I originally had just these concerns, but my mind was changed when I considered what exactly the user-interface ramifications were for people typing that 's' for 'secure' in URLs. I was convinced, and we made the change, and there have been no ill effects that I'm aware of as a result. In fact, there has been a renewed interest in Twisted for HTTP client work, because we finally made security work more or less like it's supposed to, and the standard library is so broken. I care about the health of the broader Python community, so I will passionately argue that this change should be made, but for me personally it's a lot easier to justify that everyone should use Twisted (at least since 14+) because transport security in the stdlib is such a wreck and even if it gets fixed it's going to have easy options to turn it off unilaterally so your application can never really be sure if it's getting transport security when it's requesting transport security. I completely agree with everything Glyph has said in this post. (To the shock of everyone involved I’m sure!). --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 477: selected ensurepip backports for Python 2.7
On Sep 1, 2014, at 2:22 AM, Ned Deily n...@acm.org wrote: In article cadisq7e4zachw4b_nmwewpwtxg6davndc-zwhaoajkq4sa3...@mail.gmail.com mailto:cadisq7e4zachw4b_nmwewpwtxg6davndc-zwhaoajkq4sa3...@mail.gmail.com, Nick Coghlan ncogh...@gmail.com mailto:ncogh...@gmail.com wrote: On 1 Sep 2014 09:23, Benjamin Peterson benja...@python.org wrote: On Sun, Aug 31, 2014, at 16:17, Antoine Pitrou wrote: On Mon, 1 Sep 2014 08:00:14 +1000 Nick Coghlan ncogh...@gmail.com wrote: That part of the proposal proved to be controversial, so we dropped it from the original PEP in order to focus on meeting the Python 3.4 specific release deadlines. This also had the benefit of working out the kinks in the bootstrapping processing as part of the Python 3.4 release cycle. However, we still think we should start providing pip by default to Python 2.7 users as well, at least as part of the Windows and Mac OS X installers. A much bigger than average +1 I don't agree with this. pip is simply not part of the 2.7 feature set. If you add pip to a bugfix version, then you have bugfix versions which are more featureful than others, which makes things more complicated to explain. 2.7.x has been and will be alive for so long that will already have to explain that sort thing; i.e. PEP 466 and why different bugfix releases support different versions of dependency libraries. And that is a minor complication compared with the confusion and difficulty of trying to explain to users (stuck with 2.7 for the time being) of how to install third-party packages on each platform (especially Windows) versus the simplicity of the 3.4.x story, thanks to ensurepip. Having pip available as a documented, batteries-included tool for all current releases would be a huge usability improvement. Yes this is a major driver. I mean I think I probably have an above average knowledge of how to bootstrap pip, and if you dump me on a Windows box I struggle to actually do it the first time around without stumbling around and doing things in the wrong order and the like. (Getting a compiler toolchain is worse, but yay for Wheels). Exactly. LTS is genuinely different from stopping maintenance after the next feature release - it requires considering the stability risk and user experience improvement questions separately. In this case, the problem is that the Python 2 platform *is* still evolving, but the centre of that evolution has moved to PyPI. For standard library only users, Python 2 stopped evolving back in 2010. For PyPI users, by contrast, it's still evolving at a rapid pace. For our Python 3 transition story to be coherent, we need to ensure tools like six, modernize and future are readily available, while still remaining free to evolve independently of the standard library. That means providing a package management utility as easily and as painlessly as possible. Embracing pip upstream for Python 2 as well as Python 3 also sends a powerful signal to redistributors that says your users are going to need this and makes them do additional work to *avoid* providing it. Some of them *will* choose that path, but that becomes a matter for discussion between them and their user base, rather than a problem we need to worry about upstream. FTR, I'm willing to backport the pieces I did for 3.4 and I could do the ensurepip backport, as well. I'll leave the Windows installer changes for someone else, though. Awesome, I’m of course willing to back port ensure pip itself as well. Truthfully it shouldn’t be a very difficult backport. It’s only ~200 SLOC or so and the only real things would be removing a Python3ism here or there. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On Sep 1, 2014, at 11:35 AM, Nick Coghlan ncogh...@gmail.com wrote: On 2 Sep 2014 00:59, Antoine Pitrou solip...@pitrou.net mailto:solip...@pitrou.net wrote: On Tue, 2 Sep 2014 00:53:11 +1000 Nick Coghlan ncogh...@gmail.com mailto:ncogh...@gmail.com wrote: To be frank I don't understand what you're arguing about. When I said shadowing ssl can be tricky to arrange, Chris correctly interpreted it as referring to the filesystem based privilege escalation scenario that isolated mode handles, not to normal in-process monkeypatching or module injection. There's no actual difference. You can have a sitecustomize.py that does the monkeypatching or the shadowing. There doesn't seem to be anything tricky about that. Oh, now I get what you mean - yes, sitecustomize already poses the same kind of problem as the proposed sslcustomize (hence the existence of the related command line options). I missed that you had switched to talking about using that attack vector, rather than trying to shadow stdlib modules directly through the filesystem (which is the only tricky thing I was referring to). Cheers, Nick. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io Or you can just install something with easy_install, or you can drop a .pth file and monkey patch there. You can’t stop people from overriding modules, it’s trivial to do. The sys.path ordering just makes it slightly less trivial. — Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On Sep 1, 2014, at 1:01 PM, Christian Heimes christ...@python.org wrote: On 01.09.2014 17:35, Nick Coghlan wrote: Oh, now I get what you mean - yes, sitecustomize already poses the same kind of problem as the proposed sslcustomize (hence the existence of the related command line options). If an attacker is able to place a module like sitecustomize.py in an import directory or any .pth file in a site-packages directory than this Python installation is compromised. .pth files are insidious because they are always loaded and their code is always executed. I don't see how sslcustomize is going to make a difference here. Right, this is the point I was trying to make. If you’ve installed a malicious package it’s game over. There’s nothing Python can do to help you. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On Aug 31, 2014, at 2:09 AM, Nick Coghlan ncogh...@gmail.com wrote: At the same time, we need to account for the fact that most existing organisations still trust in perimeter defence for their internal network security, and hence tolerate (or even actively encourage) the use of unsecured connections, or skipping certificate validation, internally. This is actually a really terrible idea, but it's still incredibly common due to the general failure of the technology industry to take usability issues seriously when we design security systems (at least until recently) - doing the wrong unsafe thing is genuinely easier than doing things right. Just a quick clarification in order to be a little clearer, this change will (obviously) only effect those who trust perimeter security *and* decided to install an invalid certificate instead of just using HTTP. I'm not saying that this doesn't happen, just being specific (I'm not actually sure why they would install a TLS certificate at all if they are trusting perimeter security, but I'm sure folks do). --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On Aug 31, 2014, at 5:43 PM, Christian Heimes christ...@python.org wrote: On 31.08.2014 08:09, Nick Coghlan wrote: As Antoine says here, I'm also opposed to adding more Python specific configuration options. However, I think there may be something worthwhile we can do that's closer to the way browsers work, and has the significant benefit of being implementable as a PyPI module first (more on that in a separate reply). I'm on your and Antoine's side and strictly against any additional environment variables or command line arguments. That would make the whole validation process even more complex and harder to understand. There might be a better option to give people and companies the option to tune the SSL module to their needs. Python already have a customization hook for the site module called sitecustomize. How about another module named sslcustomize? Such a module could be used to tune the ssl module to the needs of users, e.g. configure a different default context, add certificates to a default context etc. Companies could install them in a system global directory on their servers. Users could put them in their own user site directory and even each virtual env can have one sslcustomize of its own. It's fully backward compatible, doesn't add any flags and developers have the full power of Python for configuration and customization. This may be a dumb question, but why can’t sitecustomize do this already? --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On Aug 29, 2014, at 4:00 PM, M.-A. Lemburg m...@egenix.com wrote: * choice of trusted certificate: Instead of hard wiring using the system CA roots into Python it would be good to just make this default and permit the user to point Python to a different set of CA roots. This would enable using self signed certs more easily. Since these are often used for tests, demos and education, I think it's important to allow having more control of the trusted certs. If I recall OpenSSL already allows this to be configured via envvar and the python API already allows it to be configured via API. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
Sorry I was on my phone and didn’t get to fully reply to this. On Aug 29, 2014, at 4:00 PM, M.-A. Lemburg m...@egenix.com wrote: On 29.08.2014 21:47, Alex Gaynor wrote: Hi all, I've just submitted PEP 476, on enabling certificate validation by default for HTTPS clients in Python. Please have a look and let me know what you think. PEP text follows. Thanks for the PEP. I think this is generally a good idea, but some important parts are missing from the PEP: * transition plan: I think starting with warnings in Python 3.5 and going for exceptions in 3.6 would make a good transition Going straight for exceptions in 3.5 is not in line with our normal procedures for backwards incompatible changes. As far as a transition plan, I think that this is an important enough thing to have an accelerated process. If we need to provide a warning than let’s add it to the next 3.4 otherwise it’s going to be 2.5+ years until we stop being unsafe by default. Another problem with this is that I don’t think it’s actually possible to do. Python itself isn’t validating the TLS certificates, OpenSSL is doing that. To my knowledge OpenSSL doesn’t have a way to say “please validate these certificates and if they don’t validate go ahead and keep going and just let me get a warning from it”. It’s a 3 way switch, no validation, validation if a certificate is provided, and validation always. Now that’s strictly for the “verify the certificate chain” portion, the hostname verification is done entirely on our end and we could do something there… but I’m not sure it makes sense to do so if we can’t do it for invalid certificates too. * configuration: It would be good to be able to switch this on or off without having to change the code, e.g. via a command line switch and environment variable; perhaps even controlling whether or not to raise an exception or warning. I’m on the fence about this, if someone provides a certificate that we can validate against (which can be done without touching the code) then the only thing that really can’t be “fixed” without touching the code is if someone has a certificate that is otherwise invalid (expired, not yet valid, wrong hostname, etc). I’d say if I was voting on this particular thing I’d be -0, I’d rather it didn’t exist but I wouldn’t cry too much if it did. * choice of trusted certificate: Instead of hard wiring using the system CA roots into Python it would be good to just make this default and permit the user to point Python to a different set of CA roots. This would enable using self signed certs more easily. Since these are often used for tests, demos and education, I think it's important to allow having more control of the trusted certs. Like my other email said, the Python API has everything needed to easily specify your own CA roots and/or disable the validations. The OpenSSL library also allows you to specify either a directory or a file to change the root certificates without code changes. The only real problems with the APIs are that the default is bad and an unrelated thing where you can’t pass in an in memory certificate. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On Aug 29, 2014, at 5:42 PM, R. David Murray rdmur...@bitdance.com wrote: On Fri, 29 Aug 2014 17:11:35 -0400, Donald Stufft don...@stufft.io wrote: Sorry I was on my phone and didn’t get to fully reply to this. On Aug 29, 2014, at 4:00 PM, M.-A. Lemburg m...@egenix.com wrote: * configuration: It would be good to be able to switch this on or off without having to change the code, e.g. via a command line switch and environment variable; perhaps even controlling whether or not to raise an exception or warning. I’m on the fence about this, if someone provides a certificate that we can validate against (which can be done without touching the code) then the only thing that really can’t be “fixed” without touching the code is if someone has a certificate that is otherwise invalid (expired, not yet valid, wrong hostname, etc). I’d say if I was voting on this particular thing I’d be -0, I’d rather it didn’t exist but I wouldn’t cry too much if it did. Especially if you want an accelerated change, there must be a way to *easily* get back to the previous behavior, or we are going to catch a lot of flack. There may be only 7% of public certs that are problematic, but I'd be willing to bet you that there are more not-really-public ones that are critical to day to day operations *somewhere* :) wget and curl have 'ignore validation' as a command line flag for a reason. Right, that’s why I’m on the fence :) On one hand, it’s going to break things for some people, (arguably they are already broken, just silently so, but we’ll leave that argument aside) and a way to get back the old behavior is good. There are already ways within the Python code itself, so that’s covered. From outside of the Python code there are ways if the certificate is untrusted but otherwise valid which are pretty easy to do. The major “gap” is when you have an actual invalid certificate due to expiration or hostname or some other such thing. On the other hand Python is not wget/curl and the people who are most likely to be the target for a “I can’t change the code but I need to get the old behavior back” are people who are likely to not be invoking Python itself but using something written in Python which happens to be using Python. IOW they might be executing “foobar” not “python -m foobar”. Like I said though, I’m personally fine either way so don’t take this as being against that particular change! --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On Aug 29, 2014, at 5:58 PM, M.-A. Lemburg m...@egenix.com wrote: On 29.08.2014 23:11, Donald Stufft wrote: Sorry I was on my phone and didn’t get to fully reply to this. On Aug 29, 2014, at 4:00 PM, M.-A. Lemburg m...@egenix.com wrote: On 29.08.2014 21:47, Alex Gaynor wrote: Hi all, I've just submitted PEP 476, on enabling certificate validation by default for HTTPS clients in Python. Please have a look and let me know what you think. PEP text follows. Thanks for the PEP. I think this is generally a good idea, but some important parts are missing from the PEP: * transition plan: I think starting with warnings in Python 3.5 and going for exceptions in 3.6 would make a good transition Going straight for exceptions in 3.5 is not in line with our normal procedures for backwards incompatible changes. As far as a transition plan, I think that this is an important enough thing to have an accelerated process. If we need to provide a warning than let’s add it to the next 3.4 otherwise it’s going to be 2.5+ years until we stop being unsafe by default. Fine with me; we're still early in the Python 3.4 patch level releases. Another problem with this is that I don’t think it’s actually possible to do. Python itself isn’t validating the TLS certificates, OpenSSL is doing that. To my knowledge OpenSSL doesn’t have a way to say “please validate these certificates and if they don’t validate go ahead and keep going and just let me get a warning from it”. It’s a 3 way switch, no validation, validation if a certificate is provided, and validation always. Now that’s strictly for the “verify the certificate chain” portion, the hostname verification is done entirely on our end and we could do something there… but I’m not sure it makes sense to do so if we can’t do it for invalid certificates too. OpenSSL provides a callback for certificate validation, so it is possible to issue a warning and continue with accepting the certificate. Ah right, I forgot about that. I was thinking in terms of CERT_NONE, CERT_OPTIONAL, CERT_REQUIRED. I think it’s fine to add a warning if possible to Python 3.4, I just couldn’t think off the top of my head a way of doing it. * configuration: It would be good to be able to switch this on or off without having to change the code, e.g. via a command line switch and environment variable; perhaps even controlling whether or not to raise an exception or warning. I’m on the fence about this, if someone provides a certificate that we can validate against (which can be done without touching the code) then the only thing that really can’t be “fixed” without touching the code is if someone has a certificate that is otherwise invalid (expired, not yet valid, wrong hostname, etc). I’d say if I was voting on this particular thing I’d be -0, I’d rather it didn’t exist but I wouldn’t cry too much if it did. If you're testing code or trying out some new stuff, you don't want to get a valid cert first, but instead go ahead with a self signed one. That's the use case. * choice of trusted certificate: Instead of hard wiring using the system CA roots into Python it would be good to just make this default and permit the user to point Python to a different set of CA roots. This would enable using self signed certs more easily. Since these are often used for tests, demos and education, I think it's important to allow having more control of the trusted certs. Like my other email said, the Python API has everything needed to easily specify your own CA roots and/or disable the validations. The OpenSSL library also allows you to specify either a directory or a file to change the root certificates without code changes. The only real problems with the APIs are that the default is bad and an unrelated thing where you can’t pass in an in memory certificate. Are you sure that's possible ? Python doesn't load the openssl.cnf file and the SSL_CERT_FILE, SSL_CERT_DIR env vars only work for the openssl command line binary, AFAIK. I’m not 100% sure on that. I know they are not limited to the command line binary as ruby uses those environment variables in the way I described above. I do not believe that Ruby has done anything special to enable the use of those variables. It’s possible we’re doing something differently that bypasses those variables though. If that is the case then yes let’s add it, ideally doing whatever it needs to be to make OpenSSL respect those variables, or else respecting them ourselves. In any case, Python will have to tap into the OS CA root provider using special code and this code could easily be made to check other dirs/files as well. The point is that it should be possible to change this default at the Python level, without needing application code changes. Ok, I’m not opposed to it FWIW. Just sayiing I’m pretty sure those things already exist in the form
Re: [Python-Dev] PEP 467: Minor API improvements for bytes bytearray
On Aug 17, 2014, at 1:07 PM, Raymond Hettinger raymond.hettin...@gmail.com wrote: On Aug 17, 2014, at 1:41 AM, Nick Coghlan ncogh...@gmail.com mailto:ncogh...@gmail.com wrote: If I see bytearray(10) there is nothing there that suggests this creates an array of length 10 and initialises it to zero to me. I'd be more inclined to guess it would be equivalent to bytearray([10]). bytearray.zeros(10), on the other hand, is relatively clear, independently of user expectations. Zeros would have been great but that should have been done originally. The time to get API design right is at inception. Now, you're just breaking code and invalidating any published examples. Another thought is that the core devs should be very reluctant to deprecate anything we don't have to while the 2 to 3 transition is still in progress. Every new deprecation of APIs that existed in Python 2.7 just adds another obstacle to converting code. Individually, the differences are trivial. Collectively, they present a good reason to never migrate code to Python 3. This is actually one of the inconsistencies between the Python 2 and 3 binary APIs: However, bytearray(n) is the same in both Python 2 and Python 3. Changing it in Python 3 increases the gulf between the two. The further we let Python 3 diverge from Python 2, the less likely that people will convert their code and the harder you make it to write code that runs under both. FWIW, I've been teaching Python full time for three years. I cover the use of bytearray(n) in my classes and not a single person out of 3000+ engineers have had a problem with it. I seriously question the PEP's assertion that there is a real problem to be solved (i.e. that people are baffled by bytearray(bufsiz)) and that the problem is sufficiently painful to warrant the headaches that go along with API changes. The other proposal to add bytearray.byte(3) should probably be named bytearray.from_byte(3) for clarity. That said, I question whether there is actually a use case for this. I have never seen seen code that has a need to create a byte array of length one from a single integer. For the most part, the API will be easiest to learn if it matches what we do for lists and for array.array. Sorry Nick, but I think you're making the API worse instead of better. This API isn't perfect but it isn't flat-out broken either. There is some unfortunate asymmetry between bytes() and bytearray() in Python 2, but that ship has sailed. The current API for Python 3 is pretty good (though there is still a tension between wanting to be like lists and like strings both at the same time). Raymond P.S. The most important problem in the Python world now is getting Python 2 users to adopt Python 3. The core devs need to develop a strong distaste for anything that makes that problem harder. For the record I’ve had all of the problems that Nick states and I’m +1 on this change. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 467: Minor API improvements for bytes bytearray
On Aug 17, 2014, at 5:19 PM, Raymond Hettinger raymond.hettin...@gmail.com wrote: On Aug 17, 2014, at 11:33 AM, Ethan Furman et...@stoneleaf.us mailto:et...@stoneleaf.us wrote: I've had many of the problems Nick states and I'm also +1. There are two code snippets below which were taken from the standard library. Are you saying that: 1) you don't understand the code (as the pep suggests) 2) you are willing to break that code and everything like it 3) and it would be more elegantly expressed as: charmap = bytearray.zeros(256) and mapping = bytearray.zeros(256) At work, I have network engineers creating IPv4 headers and other structures with bytearrays initialized to zeros. Do you really want to break all their code? No where else in Python do we create buffers that way. Code like msg, who = s.recvfrom(256) is the norm. Also, it is unclear if you're saying that you have an actual use case for this part of the proposal? ba = bytearray.byte(65) And than the code would be better, clearer, and faster than the currently working form? ba = bytearray([65]) Does there really need to be a special case for constructing a single byte? To me, that is akin to proposing list.from_int(65) as an important special case to replace [65]. If you must muck with the ever changing bytes() API, then please leave the bytearray() API alone. I think we should show some respect for code that is currently working and is cleanly expressible in both Python 2 and Python 3. We aren't winning users with API churn. FWIW, I guessing that the differing view points in the thread stem mainly from the proponents experiences with bytes() rather than from experience with bytearray() which doesn't seem to have any usage problems in the wild. I've never seen a developer say they didn't understand what buf = bytearray(1024) means. That is not an actual problem that needs solving (or breaking). What may be an actual problem is code like char = bytes(1024) though I'm unclear what a user might have actually been trying to do with code like that. I think this is probably correct. I generally don’t think that bytes(1024) makes much sense at all, especially not as a default constructor. Most likely it exists to be similar to bytearray(). I don't have a specific problem with bytearray(1024), though I do think it's more elegantly and clearly described as bytearray.zeros(1024), but not by much. I find bytes.byte()/bytearray to be needed as long as there isn't a simple way to iterate over a bytes or bytearray in a way that yields bytes or bytearrays instead of integers. To be honest I can't think of a time when I'd actually *want* to iterate over a bytes/bytearray as integers. Although I realize there is unlikely to be a reasonable method to change that now. If iterbytes is added I'm not sure where i'd personally use either bytes.byte() or bytearray.byte(). In general though I think that overloading a single constructor method to do something conceptually different based on the type of the parameter leads to these kind of confusing scenarios and that having differently named constructors for the different concepts is far clearer. So given all that, I am: * +1 for some method of iterating over both types as bytes instead of integers. * +1 on adding .zeros to both types as an alternative and preferred method of creating a zero filled instance and deprecating the original method[1]. * -0 on adding .byte to both types as an alternative method of creating a single byte instance. * -1 On changing the meaning of bytearray(1024). * +/-0 on changing the meaning of bytes(1024), I think that bytes(1024) is likely to *not* be what someone wants and that what they really want is bytes([N]). I also think that the number one reason for someone to be doing bytes(N) is because they were attempting to iterate over a bytes or bytearray object and they got an integer. I also think that it's bad that this changes from 2.x to 3.x and I wish it hadn't. However I can't decide if it's worth reverting this at this time or not. [1] By deprecating I mean, raise a deprecation warning, or something but my thoughts on actually removing the other methods are listed explicitly. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 4000 to explicitly declare we won't be doing a Py3k style compatibility break again?
On Sun, Aug 17, 2014, at 09:02 PM, Guido van Rossum wrote: On Sun, Aug 17, 2014 at 6:29 AM, Barry Warsaw ba...@python.org wrote: On Aug 16, 2014, at 07:43 PM, Guido van Rossum wrote: (Don't understand this to mean that we should never deprecate things. Deprecations will happen, they are necessary for the evolution of any programming language. But they won't ever hurt in the way that Python 3 hurt.) It would be useful to explore what causes the most pain in the 2-3 transition? IMHO, it's not the deprecations or changes such as print - print(). It's the bytes/str split - a fundamental change to core and common data types. The question then is whether you foresee any similar looming pervasive change? [*] I'm unsure about what's the single biggest pain moving to Python 3. In the past I would have said that it's for sure the bytes/str split (which both the biggest pain and the biggest payoff). But if I look carefully into the soul of teams that are still on 2.7 (I know a few... :-), I think the real reason is that Python 3 changes so many different things, you have to actually understand your code to port it (unlike with minor version transitions, where the changes usually spike in one specific area, and you can leave the rest to normal attrition and periodic maintenance). In my experience bytes/str is the single biggest change that causes the most problems. Most of the other changes can be mechanically transformed and/or papered over using helpers like six. The bytes/str change is the main one that requires understanding code and where it requires a serious untangling of things in code bases where str/bytes are freely used intechangingbly. Often times this requires making a decision about what *should* be bytes or str as well which requires having some deep knowledge about the APIs in question too. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 467: Minor API improvements for bytes bytearray
from __future__ import bytesdoneright? :D -- Donald Stufft don...@stufft.io On Sun, Aug 17, 2014, at 09:40 PM, Antoine Pitrou wrote: Le 17/08/2014 20:08, Nick Coghlan a écrit : On 18 Aug 2014 09:57, Barry Warsaw ba...@python.org mailto:ba...@python.org wrote: On Aug 18, 2014, at 09:12 AM, Nick Coghlan wrote: I'm talking more generally - do you *really* want to be explaining that bytes behaves like a tuple of integers, while bytes.bytes behaves like a tuple of bytes? I would explain it differently though, using concrete examples. data = bytes(...) for i in data: # iterate over data as integers for i in data.bytes: # iterate over data as bytes But whatever. I just wish there was something better than iterbytes. There's actually another aspect to your idea, independent of the naming: exposing a view rather than just an iterator. So that view would actually be the bytes object done right? Funny :-) Will it have lazy slicing? Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] buildbot.python.org down again?
On Jul 8, 2014, at 12:58 AM, Nick Coghlan ncogh...@gmail.com wrote: On 7 Jul 2014 10:47, Guido van Rossum gu...@python.org wrote: It would still be nice to know who the appropriate persons are. Too much of our infrastructure seems to be maintained by house elves or the ITA. I volunteered to be the board's liaison to the infrastructure team, and getting more visibility around what the infrastructure *is* and how it's monitored and supported is going to be part of that. That will serve a couple of key purposes: - making the points of escalation clearer if anything breaks or needs improvement (although infrastruct...@python.org is a good default choice) - making the current todo list of the infrastructure team more visible (both to calibrate resolution time expectations and to provide potential contributors an idea of what's involved) Noah has already set up http://status.python.org/ to track service status, I can see about getting buildbot.python.org added to the list. Cheers, Nick. We (the infrastructure team) were actually looking earlier about buildbot.python.org and we’re not entirely sure who owns buildbot.python.org. Unfortunately a lot of the *.python.org services are in a similar state where there is no clear owner. Generally we've not wanted to just step in and take over for fear of stepping on someones toes but it appears that perhaps buildbot.p.o has no owner? - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 2.7 patch levels turning two digit
On Jun 23, 2014, at 2:09 AM, Martin v. Löwis mar...@v.loewis.de wrote: * Should we make use of the potential breakage with 2.7.10 to introduce a new Windows compiler version for Python 2.7 ? Assuming it is a good idea to continue producing Windows binaries for 2.7, I think it would be a bad idea to switch compilers. It will cause severe breakage of 2.7 installations, much more problematic than switching to two-digit version numbers. I agree with this, we’ve just finally started getting things to the point where it makes a lot of sense for binary distributions for Windows. Breaking all of them on 2.7 would be very bad. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 2.7 patch levels turning two digit
On Jun 23, 2014, at 3:27 PM, M.-A. Lemburg m...@egenix.com wrote: On 23.06.2014 18:09, Donald Stufft wrote: On Jun 23, 2014, at 2:09 AM, Martin v. Löwis mar...@v.loewis.de wrote: * Should we make use of the potential breakage with 2.7.10 to introduce a new Windows compiler version for Python 2.7 ? Assuming it is a good idea to continue producing Windows binaries for 2.7, I think it would be a bad idea to switch compilers. It will cause severe breakage of 2.7 installations, much more problematic than switching to two-digit version numbers. I agree with this, we’ve just finally started getting things to the point where it makes a lot of sense for binary distributions for Windows. Breaking all of them on 2.7 would be very bad. Err, sorry that “We” was with my pip hat on. Not sure what you mean. We've had binary wininst distributions for Windows for more than a decade, and egg and msi distributions for 8 years :-) Nonetheless, changing the compiler will not only break pip, but every automated installer tool (easy_install, buildout) that i’m aware of. The blow back for binary installation is going to be huge I think. But without access to the VS 2008 compiler that is needed to compile those extensions, it will become increasingly difficult for package authors to provide such binary packages, so we have to ask ourselves: What's worse: breaking old Windows binaries for Python 2.7 or not having updated and new Windows binaries for Python 2.7 at all in a few years ? At the risk of getting Guido to post his slide again, I still think the solution to the old compiler is to just roll a 2.8 with minimal changes. It could even be a good place to move to the ssl backport changes too since they were the riskier set of changes in PEP466. But either way, if a compiler does change in a 2.7 release we’ll need to update a lot of tooling to cope with that, so any plan to do that should include that and a timeline for adoption of that. Switching to a newer compiler will make things easier for everyone and we'd see more binary packages for Windows again. Given that Python 2.7 support was extended for another 5 years at the recent Python Language Summit to 2020, we have to face this breakage sooner or later anyway. Extended support for VS 2008 will end in 2018 (but then: Python developers usually don't have extended support contracts with MS). Service pack support has already ended in 2009. Depending on how you see it, using such an old compiler also poses security risks. The last security update for VS 2008 dates back to 2011 (http://support.microsoft.com/kb/2538243). -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jun 23 2014) Python Projects, Consulting and Support ... http://www.egenix.com/ mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2014-06-17: Released eGenix PyRun 2.0.0 ... http://egenix.com/go58 2014-07-02: Python Meeting Duesseldorf ... 9 days to go 2014-07-21: EuroPython 2014, Berlin, Germany ... 28 days to go eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 2.7 patch levels turning two digit
On Jun 23, 2014, at 4:31 PM, Barry Warsaw ba...@python.org wrote: On Jun 23, 2014, at 04:20 PM, Donald Stufft wrote: At the risk of getting Guido to post his slide again, I still think the solution to the old compiler is to just roll a 2.8 with minimal changes. No. It's not going to happen, for all the reasons discussed previously. Python 2.8 is not a solution to anything. If a new, incompatible compiler suite is required, why can't there just be multiple Windows downloads on https://www.python.org/download/releases/2.7.7/ ? Well, on reason is that you'd have to convince MvL or someone else to take over the work that would require, but that's gotta be *much* lighter weight than releasing a Python 2.8. As far as I am aware, a 2.7 with a different compiler, even if it’s just an option is an attractive nuisance. None of the tooling right now differentiates between binary compatibility by anything other than “CPython 2.7”. The end result of having a 2.7 which is built with the old compiler, and a 2.7 built with the new compiler is that you’ll end up with binary distributions which work sometimes if you’re lucky and the creator of the binary distribution and you happened to pick the same “variant” of 2.7. Most likely result is all the binary distributions will *mostly* still depend on using the old compiler because of the corpus of existing binary packages that depend on that. Which means that the 2.7 with new compiler will exist entirely to act as a footgun to anyone who picks it and also wants to use binary packages. -Barry ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 2.7 patch levels turning two digit
On Jun 23, 2014, at 4:31 PM, Martin v. Löwis mar...@v.loewis.de wrote: Would that mitigate the pain, assuming that Steve (or someone else) would be willing to build the additional installers for the transition period? I've done something similar on a smaller scale with the OS X 32-bit installer for 2.7.x but that impact is much less as the audience for that installer is much smaller. Well, the question really is whether precompiled extension modules available from PyPI would work on both compilers. I understand that for OSX, you typically don't have precompiled binaries for the extension modules, so installation compiles the modules from scratch. This is easier, as it can use the ABI of the Python which will be installed to. If you go the parallel ABIs route, extension authors have to provide two parallel sets of packages as well. Given 32-bit and 64-bit packages, this will make actually two additional packages - just as if they had to support another Python version. As far as I know, stuff on OSX is generally built for “X compiler or later” so binary compatibility is kept as long as you’re using an “or later” but I could be wrong about that. Using binary packages on OSX is a much less frequent thing I think though since getting a working compiler toolchain is easier there. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 2.7 patch levels turning two digit
On Jun 23, 2014, at 5:07 PM, M.-A. Lemburg m...@egenix.com wrote: On 23.06.2014 22:20, Donald Stufft wrote: On Jun 23, 2014, at 3:27 PM, M.-A. Lemburg m...@egenix.com wrote: On 23.06.2014 18:09, Donald Stufft wrote: On Jun 23, 2014, at 2:09 AM, Martin v. Löwis mar...@v.loewis.de wrote: * Should we make use of the potential breakage with 2.7.10 to introduce a new Windows compiler version for Python 2.7 ? Assuming it is a good idea to continue producing Windows binaries for 2.7, I think it would be a bad idea to switch compilers. It will cause severe breakage of 2.7 installations, much more problematic than switching to two-digit version numbers. I agree with this, we’ve just finally started getting things to the point where it makes a lot of sense for binary distributions for Windows. Breaking all of them on 2.7 would be very bad. Err, sorry that “We” was with my pip hat on. Not sure what you mean. We've had binary wininst distributions for Windows for more than a decade, and egg and msi distributions for 8 years :-) Nonetheless, changing the compiler will not only break pip, but every automated installer tool (easy_install, buildout) that i’m aware of. The blow back for binary installation is going to be huge I think. But without access to the VS 2008 compiler that is needed to compile those extensions, it will become increasingly difficult for package authors to provide such binary packages, so we have to ask ourselves: What's worse: breaking old Windows binaries for Python 2.7 or not having updated and new Windows binaries for Python 2.7 at all in a few years ? At the risk of getting Guido to post his slide again, I still think the solution to the old compiler is to just roll a 2.8 with minimal changes. It could even be a good place to move to the ssl backport changes too since they were the riskier set of changes in PEP466. But either way, if a compiler does change in a 2.7 release we’ll need to update a lot of tooling to cope with that, so any plan to do that should include that and a timeline for adoption of that. Sure, and we'd need to hash out possible solutions to minimize breakage, but first we'll have to see whether we want to consider this step or not. BTW: It's strange that I'm arguing for breaking things. I'm usually on the other side of such arguments :-) Ok. I’m just making sure that any proposal to do this includes how it plans to work around/minimize that. I agree with Martin (I think) that trying to fix the entire ecosystem to cope with that change is going to be far more work than folks realize and that it needs to be an explicit part of the discussion when deciding how to solve the problem. Normally when I see someone suggest that switching compilers in 2.7.x is likely to be less work than releasing a 2.8 It normally appears to me they haven’t looked at the impact on the packaging tooling. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 2.7 patch levels turning two digit
On Jun 23, 2014, at 5:22 PM, Barry Warsaw ba...@python.org wrote: On Jun 23, 2014, at 05:15 PM, Donald Stufft wrote: Normally when I see someone suggest that switching compilers in 2.7.x is likely to be less work than releasing a 2.8 It normally appears to me they haven’t looked at the impact on the packaging tooling. Just to be clear, releasing a Python 2.8 has enormous impact outside of just the amount of work to do it. It's an exceedingly bad idea. Can you clarify? Also FWIW I’m not really married to the 2.8 thing, it’s mostly that, on Windows, the X.Y release prior to the ABI thing in 3.x _was_ the ABI so all the tooling builds on that. So you need to either 1) Stick with the old Compiler 2) Release 2.8 3) Do all the work to fix all the tooling to cope with the fact that X.Y isn’t the ABI on 2.x anymore I don’t think a reasonable option is: 4) Just switch compilers and leave it on someone else’s doorsteps to fix the entire packaging tool chain to cope. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 2.7 patch levels turning two digit
On Jun 23, 2014, at 5:48 PM, Chris Angelico ros...@gmail.com wrote: On Tue, Jun 24, 2014 at 6:42 AM, Martin v. Löwis mar...@v.loewis.de wrote: See my other message. It's actually heavier, since it requires changes to distutils, PyPI, pip, buildout etc., all which know how to deal with Python minor version numbers, but are unaware of the notion of competing ABIs on Windows (except that they know how to deal with 32-bit vs. 64-bit). Is it possible to hijack the deal with 32-bit vs 64-bitness of things to handle the different compilers? So, for instance, there might be a 32-bit-NewCompiler and a 64-bit-NewCompiler, two new architectures, just as if someone came out with a 128-bit Windows and built Python 2.7 for it. Would packaging be able to handle that more easily than a compiler change within the same architecture? ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io I’m not sure about this FWIW. I’d have to look at the implementations of stuff to see how they’d cope with a new thing like that. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 2.7 patch levels turning two digit
-02: Python Meeting Duesseldorf ... 11 days to go eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
On Jun 6, 2014, at 11:41 AM, Steve Dower steve.do...@microsoft.com wrote: words +1 from me. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
On Jun 6, 2014, at 3:04 PM, Brian Curtin br...@python.org wrote: On Fri, Jun 6, 2014 at 10:56 PM, dw+python-...@hmmz.org wrote: On Fri, Jun 06, 2014 at 10:49:24PM +0400, Brian Curtin wrote: None of the options are particularly good, but yes, I think that's an option we have to consider. We're supporting 2.7.x for 6 more years on a compiler that is already 6 years old. Surely that is infinitely less desirable than simply bumping the minor version? It's definitely not desirable, but simply bumping the minor version is not A Thing. Why? I mean even if it’s the same thing as 2.7 just with an updated compiler that seems like a better answer than having to deal with 2.7.whatever suddenly breaking all C exts. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
On Jun 6, 2014, at 3:09 PM, Brian Curtin br...@python.org wrote: On Fri, Jun 6, 2014 at 11:08 PM, Donald Stufft don...@stufft.io wrote: On Jun 6, 2014, at 3:04 PM, Brian Curtin br...@python.org wrote: On Fri, Jun 6, 2014 at 10:56 PM, dw+python-...@hmmz.org wrote: On Fri, Jun 06, 2014 at 10:49:24PM +0400, Brian Curtin wrote: None of the options are particularly good, but yes, I think that's an option we have to consider. We're supporting 2.7.x for 6 more years on a compiler that is already 6 years old. Surely that is infinitely less desirable than simply bumping the minor version? It's definitely not desirable, but simply bumping the minor version is not A Thing. Why? I mean even if it’s the same thing as 2.7 just with an updated compiler that seems like a better answer than having to deal with 2.7.whatever suddenly breaking all C exts. Because then we have to maintain 2.8 at a time when no one even wants to maintain 2.7? Is it really any difference in maintenance if you just stop applying updates to 2.7 and switch to 2.8? If 2.8 is really just 2.7 with a new compiler then there should be no functional difference between doing that and doing a 2.7.whatever except all of the tooling that relies on the compiler not to change in micro releases won’t suddenly break and freak out. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
On Jun 6, 2014, at 3:33 PM, Chris Angelico ros...@gmail.com wrote: On Sat, Jun 7, 2014 at 5:11 AM, Donald Stufft don...@stufft.io wrote: Is it really any difference in maintenance if you just stop applying updates to 2.7 and switch to 2.8? If 2.8 is really just 2.7 with a new compiler then there should be no functional difference between doing that and doing a 2.7.whatever except all of the tooling that relies on the compiler not to change in micro releases won’t suddenly break and freak out. If the only difference between 2.7 and 2.8 is the compiler used on Windows, what happens on Linux and other platforms? A Python 2.8 would have to be materially different from Python 2.7, not just binarily incompatible on one platform. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io Well it’d contain bug fixes and whatever other sorts of things you’d put into a 2.7.whatever release. So they’d still want to upgrade to 2.8 since that’ll have bug fixes. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
On Jun 6, 2014, at 9:05 PM, Terry Reedy tjre...@udel.edu wrote: On 6/6/2014 6:47 PM, Nathaniel Smith wrote: On Fri, Jun 6, 2014 at 11:43 PM, Sturla Molden sturla.mol...@gmail.com wrote: Brett Cannon bcan...@gmail.com wrote: Nope. A new minor release of Python is a massive undertaking which is why we have saved ourselves the hassle of doing a Python 2.8 or not giving a clear signal as to when Python 2.x will end as a language. Why not just define Python 2.8 as Python 2.7 except with a newer compiler? I cannot see why that would be massive undertaking, if changing compiler for 2.7 is neccesary anyway. This would require recompiling all packages on OS X and Linux, even though nothing had changed. If you are suggesting that a Windows compiler change should be invisible to non-Windows users, I agree. Let us assume that /pcbuild remains for those who have vc2008 and that /pcbuild14 is added (and everything else remains as is). Then the only other thing that would change is the Windows installer released on Python.org. Call than 2.7.9W or whatever on the download site and interactive startup message to signal that something is different. -- Terry Jan Reedy ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io How are packaging tools supposed to cope with this? AFAIK there is nothing in most of them to deal with a X.Y.Z release suddenly dealing with a different compiler. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
On Jun 7, 2014, at 12:41 AM, Nick Coghlan ncogh...@gmail.com wrote: On 7 June 2014 08:43, Sturla Molden sturla.mol...@gmail.com wrote: Brett Cannon bcan...@gmail.com wrote: Nope. A new minor release of Python is a massive undertaking which is why we have saved ourselves the hassle of doing a Python 2.8 or not giving a clear signal as to when Python 2.x will end as a language. Why not just define Python 2.8 as Python 2.7 except with a newer compiler? I cannot see why that would be massive undertaking, if changing compiler for 2.7 is neccesary anyway. It's honestly astonishing the number of people that tell us doing a new minor release of Python 2 is easy, and then refuse to believe us when we tell them it isn't. It's 2014 and Python *2.7*, which was released in *2010*, is STILL BEING ROLLED OUT. One part of the rollout that is near dear to my own heart is the fact that Red Hat Enterprise Linux 7 and CentOS 7 are still in their respective release candidate phases, and it is the 6 - 7 transition that finally upgrades their system Pythons from 2.6 to 2.7. Maya 2014 MotionBuilder 2014 are also the first versions Autodesk are shipping that use 2.7 rather than 2.6 as the scripting engine (although my understanding is that Autodesk don't guarantee compatibility with Python C extensions that aren't built specifically for use with their products, so they already use a newer C runtime on Windows than we do). And once those two dominoes fall, then there'll be some additional follow on upgrade work in some parts of the developer community as the *users* that receive their Python through those channels rather than directly from upstream switch from 2.6 to 2.7 and stumble over the small compatibility breaks between those two releases. Words like just, or simple, or easy really have no place being applied to a task where the time required to fully execute it with *no significant problems* is still measured in years. How much of that time exists because there were actual significant changes from 2.6 to 2.7 and how much of it would not need to exist if 2.8 was literally 2.7.Z with a new compiler on Windows. IOW is it the *version* number that causes the slow upgrade, or is it the fact that there are enough changes that it can’t be safely applied automatically. That said, there are definitely problems with toolchain availability on Windows for Python 2, and it isn't clear yet how that will be addressed in the long run. Steve is working on ensuring the official toolchain and C runtime binaries are more readily available from MS. Other folks are independently looking into ensuring that open source toolchains (like mingw) can be used effectively to at least build Python C extensions for Windows (and ironing out some of the glitches with that approach that others have mentioned). The Python Packaging Authority are continuing to work on the wheel based infrastructure to help avoid end users having to compile anything in the first place, and redistributors like ActiveState, Enthought Continuum Analytics also make it possible for many end users to just ignore these upstream concerns. An extension compatibility break would be an absolutely last resort, pursued only if all other attempts at resolving the challenges have demonstrably failed - even at the best of times it can take months for C extension authors to start publishing compatible binaries for a new minor release, so we'd have to assume that time would be even longer for a Python 2.7 maintenance release, if they published updated binaries at all. Regards, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
On Jun 7, 2014, at 12:58 AM, Nick Coghlan ncogh...@gmail.com wrote: On 7 June 2014 14:47, Donald Stufft don...@stufft.io wrote: On Jun 7, 2014, at 12:41 AM, Nick Coghlan ncogh...@gmail.com wrote: Words like just, or simple, or easy really have no place being applied to a task where the time required to fully execute it with *no significant problems* is still measured in years. How much of that time exists because there were actual significant changes from 2.6 to 2.7 and how much of it would not need to exist if 2.8 was literally 2.7.Z with a new compiler on Windows. IOW is it the *version* number that causes the slow upgrade, or is it the fact that there are enough changes that it can’t be safely applied automatically. It's the version number change itself. Python 2.7 was covered by the language moratorium, so it consists almost entirely of standard library changes, and the porting notes are minimal: https://docs.python.org/2/whatsnew/2.7.html#porting-to-python-2-7 I’m not sure I agree, the porting docs only show a subset of changes, you also have a lot of new stuff like OrderedDict, dict comprehensions, set literals, argparse, dict views, memory views, etc. AFAIK stable releases don’t jump versions because all of these new features are risks, not because a number didn’t change. I don’t particularly care too much though, I just think that bumping the compiler in a 2.7.Z release is a really bad idea and that either of the other two options are massively better. We didn't even switch compilers on Windows (both 2.6 and 2.7 use VS 2008). I can't think of a better demonstration than the slow pace of the Python 2.7 rollout that the challenges with doing a new minor release of Python really aren't technical ones at the language level - they're technical and administrative challenges in the way the language version number interacts with the broader Python ecosystem, especially the various redistribution channels. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Internal representation of strings and Micropython
I think UTF8 is the best option. On Jun 3, 2014, at 9:17 PM, Steven D'Aprano st...@pearwood.info wrote: There is a discussion over at MicroPython about the internal representation of Unicode strings. Micropython is aimed at embedded devices, and so minimizing memory use is important, possibly even more important than performance. (I'm not speaking on their behalf, just commenting as an interested outsider.) At the moment, their Unicode support is patchy. They are talking about either: * Having a build-time option to restrict all strings to ASCII-only. (I think what they mean by that is that strings will be like Python 2 strings, ASCII-plus-arbitrary-bytes, not actually ASCII.) * Implementing Unicode internally as UTF-8, and giving up O(1) indexing operations. https://github.com/micropython/micropython/issues/657 Would either of these trade-offs be acceptable while still claiming Python 3.4 compatibility? My own feeling is that O(1) string indexing operations are a quality of implementation issue, not a deal breaker to call it a Python. I can't see any requirement in the docs that str[n] must take O(1) time, but perhaps I have missed something. -- Steven ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 2.7.7 and PEP 466
On May 18, 2014, at 9:53 PM, Guido van Rossum gu...@python.org wrote: On Sun, May 18, 2014 at 5:49 PM, Benjamin Peterson benja...@python.org wrote: Greetings Python users, Python 2.7.7 release candidate 1 is now available for download. [...] http://hg.python.org/cpython/raw-file/e32e3a9f3902/Misc/NEWS So what became of PEP 466? This Misc/NEWS only mentions hmac.compare_digest. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io The SSL changes were too large to get done before 2.7.7 The pbkdf2 has a patch sitting on the tracker (http://bugs.python.org/issue21304) Alex wanted someone to review before commit. I looked over it but I don’t feel strong enough in C code to call it a proper review. The guaranteed_algorithms bug depends on the pbkdf2 bug (http://bugs.python.org/issue21307) The os.urandom change had some argument and some concern that the related change in 3.4 was still new and had some bugs being ironed out so it was punted until 2.7.8 (http://bugs.python.org/issue21305) And that was everything from PEP 466. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 2.7.7 and PEP 466
Well I believe Alex did what he did during his work day @ Rackspace. Distros specifically I don’t believe so, although both Alex and myself agreed that it made sense for the SSL changes to wait until after the other changes since it was the largest and most complicated. I think that’s the one where it makes the most sense to try and garner help from Red Hat or the like. Perhaps Nick knows someone at Red Hat we can poke to see if they’d be willing to do the SSL patch :) On May 18, 2014, at 10:28 PM, Guido van Rossum gu...@python.org wrote: Thanks for the update, Donald. Did anyone get any help from Red Hat or other distros? On Sun, May 18, 2014 at 7:02 PM, Donald Stufft don...@stufft.io wrote: On May 18, 2014, at 9:53 PM, Guido van Rossum gu...@python.org wrote: On Sun, May 18, 2014 at 5:49 PM, Benjamin Peterson benja...@python.org wrote: Greetings Python users, Python 2.7.7 release candidate 1 is now available for download. [...] http://hg.python.org/cpython/raw-file/e32e3a9f3902/Misc/NEWS So what became of PEP 466? This Misc/NEWS only mentions hmac.compare_digest. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io The SSL changes were too large to get done before 2.7.7 The pbkdf2 has a patch sitting on the tracker (http://bugs.python.org/issue21304) Alex wanted someone to review before commit. I looked over it but I don’t feel strong enough in C code to call it a proper review. The guaranteed_algorithms bug depends on the pbkdf2 bug (http://bugs.python.org/issue21307) The os.urandom change had some argument and some concern that the related change in 3.4 was still new and had some bugs being ironed out so it was punted until 2.7.8 (http://bugs.python.org/issue21305) And that was everything from PEP 466. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -- --Guido van Rossum (python.org/~guido) - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython: Remove the redundant and poorly worded warning message.
On May 10, 2014, at 6:10 PM, Nick Coghlan ncogh...@gmail.com wrote: On 11 May 2014 07:37, Raymond Hettinger raymond.hettin...@gmail.com wrote: On May 10, 2014, at 2:18 PM, Alex Gaynor alex.gay...@gmail.com wrote: I think this change is a considerable usability regression for the documentation. Right now the warnings about CSPRNGs are hidden in the introductory paragraph, which users are likely to skip In the past couple of years, we've grown an unfortunate tendency to fill the docs with big warning boxes (the subprocess docs are an example of implicitly communicating that the module is dangerous and unusable). The preferred form of documentation is to be affirmatively worded, telling how to use a tool correctly and what its known limitations are. We save the warnings for cases of actual danger where otherwise well informed users get tripped-up. Tim Peters used to be around to articulate the principle that we don't write the docs with the assumption that the users are less bright than we are or that they can't read. That assumption has changed somewhat, as many users are now getting their education in programming (and how computers work in general) from introductory community workshops and the Python documentation. This means that writing our docs based on the assumption that the reader is already going to be a professional programmer is no longer adequate. This is especially essential in security related areas, as even professional programmers usually aren't sufficiently paranoid about all the ways their software can be attacked. As Alex notes, the short term way to eliminate the duplication here is to keep the security warnings and drop the material from the introductory paragraph, not go back to expecting readers to have already been alerted to randomness related cryptographic security issues in some other context. Readers that are already familiar with the security concerns will hopefully recognise that they're not a Python specific problem (and may even be appreciative of our attempts to convey the relevant knowledge to a broader audience), while readers that aren't yet aware of them may be more likely to account for them appropriately when writing their software. It's as much about cultivating a more paranoid more mindset in developers in general as it is about the contents of the specific security warning. A Security Considerations section in the module documentation can be a better way to tackle this than trying to jam everything into one warning box, but there should still be a Here be dragons warning early on noting that random *is* a potentially security sensitive module in a cryptographic context. I completely agree with Alex, Antoine, and Nick here. I’m both an experienced Python programmer and someone who is generally aware of the security implications of various parts of software. However I appreciate when I look at documentation that explicitly calls out the ways I might screw it up, especially in a security sensitive context, but I appreciate it any context really. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] python process creation overhead
On May 10, 2014, at 5:46 PM, Victor Stinner victor.stin...@gmail.com wrote: Le 10 mai 2014 22:51, Gregory Szorc gregory.sz...@gmail.com a écrit : Furthermore, Python 3 appears to be 50% slower than Python 2. Please mention the minor version. It looks like you compared 2.7 and 3.3. Please test 3.4, we made interesting progress on the startup time. There is still something to do, especially on OS X. Depending on the OS, different modules are loaded and some functions are implemented differently. For what it's worth pip is the same way, about half of our test suite involves invoking (multiple) python processes. This has historically be really slow (~30 minutes to run ~200 tests of that type). We've been able to get the wall clock run time down by parallelizing these but the sequential time is still really slow. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] python process creation overhead
Yes right, sorry I didn’t mean to imply that all that time was spent in the Python start up time. I’ve personally never actually spent time to figure out which part of that was slow because getting visibility inside of a subprocess.Popen is a pain and I’m slowly trying to rewrite our tests to not require subprocesses at all. I was only trying to chime in with a me too here about slow subprocess tests since our test suite ends up starting a couple thousand subprocesses as well and those tests are definitely our slowest. On May 10, 2014, at 6:50 PM, Guido van Rossum gu...@python.org wrote: Yeah, but 200 test in 30 minutes is 9 *seconds* per test -- the Python startup time is only a tiny fraction of that (20-40 *milliseconds*). On Sat, May 10, 2014 at 3:33 PM, Donald Stufft don...@stufft.io wrote: On May 10, 2014, at 5:46 PM, Victor Stinner victor.stin...@gmail.com wrote: Le 10 mai 2014 22:51, Gregory Szorc gregory.sz...@gmail.com a écrit : Furthermore, Python 3 appears to be 50% slower than Python 2. Please mention the minor version. It looks like you compared 2.7 and 3.3. Please test 3.4, we made interesting progress on the startup time. There is still something to do, especially on OS X. Depending on the OS, different modules are loaded and some functions are implemented differently. For what it's worth pip is the same way, about half of our test suite involves invoking (multiple) python processes. This has historically be really slow (~30 minutes to run ~200 tests of that type). We've been able to get the wall clock run time down by parallelizing these but the sequential time is still really slow. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (python.org/~guido) - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On May 9, 2014, at 4:12 AM, M.-A. Lemburg m...@egenix.com wrote: On 08.05.2014 23:22, Donald Stufft wrote: On a personal note, I'm uncomfortable with the way this change is perceived as a case of *pip* enforcing a behaviour that the pip developers feel should be required. I actually don't like this change particularly. So having pip implement the behaviour required by that PEP is to me simply a case of compliance with the agreed standard. But now, as a pip developer, being held responsible for the resulting user pain, and being expected to defend it, does not make me happy. I think the pain is being overrepresented and the positives are being ignored. The problem is the benefits of this PEP are much like the benefits of TLS too. For the vast majority of people they don’t notice anything different except installing things is faster and more reliable. They don’t attribute that to the PEP or this decision, they just internalize it as the new norm. However the people who this does affect will seek out why it broke and raise an issue citing that thing specifically. This creates a perception of lots of pain for no gain when the reality is not that. Donald: I don't think anyone is arguing that hosting packages on PyPI is a bad thing and PyPI as a service has gotten a lot better than it was a few years ago. Didn’t mean to imply that I thought it was otherwise. However, I find it troubling that we as Python developers are *forcing* the whole Python world to put their code into PyPI. Forcing is a strong word. As of right now we’re forcing you to put it onto PyPI if you want to install it without *any* flags to pip. You're more then capable of hosting it externally if you wish, and pip will even tell the people who try to install it what they need to enable in order to install it. It even allows people to say I don't care about any of this crap, just make all the external stuff work. Even if pip removed the external link handling all together and required you to do something like: $ pip install --extra-index-url https://example.com/our-packages/ foo $ pip install --find-links https://example.com/our-packages/ foo We still wouldn't be forcing anyone to upload things to PyPI. We are, however, discouraging people from not hosting on PyPI and providing incentives to doing that. There are plenty good reasons not to do this, and sometimes it's even impossible if you want to stay legal (see the PEP for details). I re-read the reasons, I'm not sure I really agree with most (or all?) of them however if people really want to do it, then there is nothing stopping them. It's their choice and people on the *other* side of that who are installing these packages also get to make a choice if they want to allow it or not. Accordingly, we should respect those reasons make it possible for Python packages to live elsewhere, without having our tools put those packages into a bad light or making it harder for Python users to install such packages than needed. I'm not sure what put it into a bad light is supposed to mean here. Is it about the warning? I've already removed that completely. As far as making it harder than it needs to be, that's a hard line to draw. I personally know people for whom things that are hosted externally have caused them a lot of pain to the point where they have to automatically spider PyPI for their own dependencies every hour or so in order to isolate themselves from the failure of random dependencies suddenly no longer being installable. This breakage is going to happen for basically any project that installs their dependencies with any frequency. One of the big projects that I'm aware of that has had this problem is Openstack who, in their CI, installs things several hundred times a day. If a service they depend on goes down that causes significant disruption for them. Now they've solved it by the above solution of hosting their own mirror of PyPI that includes spidering for externally installable things. They have one or two packages left that don't host directly on PyPI at which point they'll no longer need that afaik. I don't think telling every downstream of us of any size that in order to get reliable installs they are going to have to work around PyPI, and in some cases even develop custom software in order to do so effectively, is a very user friendly position. With the checksum uploaded to PyPI, the only argument against fetching packages from other places on the Internet is reliability, not security. I've never said differently. There are some minor privacy things in there but primarily it's about reliability and that people should generally be aware where their packages are coming from. PyPI is not the only reliable file hosting system on the Internet, so this argument is rather weak. It actually doesn't matter how reliable the other systems are. Reliability wise the best possible outcome is the external host has 100
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On May 9, 2014, at 5:01 AM, Paul Moore p.f.mo...@gmail.com wrote: On 9 May 2014 05:34, Donald Stufft don...@stufft.io wrote: On May 8, 2014, at 5:22 PM, Donald Stufft don...@stufft.io wrote: Socially, this change does not seem to be having the effect of persuading more package developers to host on PyPI. The stick doesn't appear to have worked, maybe we should be trying to find a carrot? Do you have any data to point to that says it hasn’t worked? Just to see what impact it has had, I’m running my scripts again that I ran a year ago to see what has changed, already I can see they are processing MUCH faster than last year. The data has finished processing, it represents a time diff of approximately one year. The pip release that caused all of this was released about 4-5 months ago. Overall PyPI has seen a 50% growth in installable projects in that time. If the change would have had no effect we'd expect to see a ~50% increase across the board. However what we've seen is a a 60% (+10% of expected) increase in projects that can only be installed from PyPI and a 12% decrease in projects that have any unsafe files (-62% of expected). Donald, Thanks for taking the time to get those figures. It does appear that there are less cases that would be affected than the number of complaints would imply. Of course, I don’t like making claims without backing them up if I can :) The only concern I have about this type of analysis is that it doesn't weight projects. It may be (and again, I have no data to back this up) that the projects that are affected detrimentally by this change are unusually popular or otherwise significant. There's obviously no way to assess this sensibly other than by making a judgement on the level of complaints. Yea, I don’t have a good way to weight those projects in any way. Normally I could get some sort of estimate by looking at the download numbers from PyPI but well ;) For the record, here’s the list of projects that are hosted *only* safely externally or that have *any* safely externally hosted files: https://gist.github.com/dstufft/1b16c305f97fff6cef2f Most of these don’t stand out to me at all. The only ones that do are: * pyOpenSSL which has one older release that is hosted that way * argparse which has the latest release hosted this way but has older releases hosted on PyPI * new relic which only hosts older releases externally * beautifulsoup4 which hosts things safely externally *and* on PyPI * Paste which has one “external” thing which is actually only external because it used a cheeseshop.python.org link instead of a pypi.python.org link. * ipython which has one older release hosted safely externally but the latest is on PyPI * netifaces which has one older release hosted safely externally but the rest are on PyPI But arguing numbers was never my intention here, so let's just say that I concede that the change has had a positive effect, which is great. Paul I didn’t mean to try to imply that it was :) I just wanted to make sure that *my* claims were true, or if they weren’t I wanted to be able to say that I was wrong. Since I had the numbers computed already it didn’t make any sense not to share them here. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On May 9, 2014, at 7:55 AM, Paul Moore p.f.mo...@gmail.com wrote: On 9 May 2014 12:44, Donald Stufft don...@stufft.io wrote: We still wouldn't be forcing anyone to upload things to PyPI. We are, however, discouraging people from not hosting on PyPI and providing incentives to doing that. But you're doing so by inflicting pain on people using pip to install those packages. Those users complain about *pip*, not about the packages. Better to directly impact the package maintainers, rather than their users (who are innocent victims). Better still of course to encourage people to improve things, not to punish them for not doing so. We can’t directly impact the package maintainers and the vast bulk of people who have had a problem who have complained about it to pip also need to add the —allow-unverifiable flag and would not simply be able to be fixed by allowing safely externally hosted files. Looking at the numbers and what packages are hosted externally, allowing safely externally hosted files would have practically no benefit to pip’s end users. The only case that I can see with a quick scan would be it would allow the latest version of argparse. TBH I think the biggest source of confusion reduction would be to remove the “safely externally hosted’ category all together and just make it hosted on PyPI - Install by default, hosted off PyPI - requires a per package flag. However I’m sure the vocal minority would have a problem with that. I think it's important to point out that one of the driving factors that caused me to finally push for changes and what lead to PEP438 being created was that Mercurial's external hosted was being extremely flaky. I can't remember the exact details but I want to say that over the span of a week or two I was getting massive numbers of users complaining that ``pip install Mercurial`` was suddenly failing. This isn't to knock on the Mercurial folks or anything but to simply point out that these problems aren't things that just happen to (under|un)maintained software nor are they hypothetical. This PEP was born of the frustration that was being relayed to me by end users of PyPI/pip. So now pip install Mercurial always fails? And adding a flag allows it to work as well as before, but no better? How did that fix the issue? Seriously - I'm missing something here. No, This caused Mercurial to upload their packages to PyPI. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On May 9, 2014, at 8:21 AM, Paul Moore p.f.mo...@gmail.com wrote: On 9 May 2014 13:06, Donald Stufft don...@stufft.io wrote: I think it's important to point out that one of the driving factors that caused me to finally push for changes and what lead to PEP438 being created was that Mercurial's external hosted was being extremely flaky. I can't remember the exact details but I want to say that over the span of a week or two I was getting massive numbers of users complaining that ``pip install Mercurial`` was suddenly failing. This isn't to knock on the Mercurial folks or anything but to simply point out that these problems aren't things that just happen to (under|un)maintained software nor are they hypothetical. This PEP was born of the frustration that was being relayed to me by end users of PyPI/pip. So now pip install Mercurial always fails? And adding a flag allows it to work as well as before, but no better? How did that fix the issue? Seriously - I'm missing something here. No, This caused Mercurial to upload their packages to PyPI. You're claiming that Mercurial moved to hosting on PyPI solely because users suddenly needed to add a flag to install from pip? As opposed to because PyPI gave them a more reliable hosting platform, for example? OK. I certainly can't give any evidence to dispute that claim, although I'm surprised. Paul I don’t know that for a fact but If my memory is correct that’s a reasonable assumption based on the timeline. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On May 9, 2014, at 9:58 AM, M.-A. Lemburg m...@egenix.com wrote: On 09.05.2014 13:44, Donald Stufft wrote: On May 9, 2014, at 4:12 AM, M.-A. Lemburg m...@egenix.com wrote: Donald: I don't think anyone is arguing that hosting packages on PyPI is a bad thing and PyPI as a service has gotten a lot better than it was a few years ago. Didn’t mean to imply that I thought it was otherwise. However, I find it troubling that we as Python developers are *forcing* the whole Python world to put their code into PyPI. Forcing is a strong word. As of right now we’re forcing you to put it onto PyPI if you want to install it without *any* flags to pip. Which is what most people expect to be able to do. Sure, but sometimes it’s better to make an informed choice about things being installed instead of having it be installed by default and sometimes it’s better to disallow doing something at all. Further most people don’t expect an install to touch any server host other than PyPI. As far as I’m aware none of the other package repositories feature this, and even with the fact we support it barely anyone even cares to utilize it. You're more then capable of hosting it externally if you wish, and pip will even tell the people who try to install it what they need to enable in order to install it. It even allows people to say I don't care about any of this crap, just make all the external stuff work. Even if pip removed the external link handling all together and required you to do something like: $ pip install --extra-index-url https://example.com/our-packages/ foo $ pip install --find-links https://example.com/our-packages/ foo We still wouldn't be forcing anyone to upload things to PyPI. We are, however, discouraging people from not hosting on PyPI and providing incentives to doing that. This is all true, but in reality, you are making externally hosted packages second class citizens in Python land by requiring extra options even for package listings that are perfectly safe to download from other servers. As mentioned before: I can understand that you want to make downloads safe for users, but if a package is hosted on some other reliable servers and pip can check that it's a valid package, there's little to argue for not allowing such downloads. Except the fact that the only people I’ve ever seen *happy* that people can host packages externally are a handful (5) of people (tbh primarily you and Stefan) and the feedback I get from every other person around the web in unequivocally yes please get rid of externally hosted files, they’ve done nothing but cause me pain. It’s not reasonable to allow a minority of users to negatively impact the majority. There are plenty good reasons not to do this, and sometimes it's even impossible if you want to stay legal (see the PEP for details). I re-read the reasons, I'm not sure I really agree with most (or all?) of them however if people really want to do it, then there is nothing stopping them. It's their choice and people on the *other* side of that who are installing these packages also get to make a choice if they want to allow it or not. You don't have to agree with those reasons. Fact is, they exist, prevent package authors from uploading to PyPI and we as Python developers should respect those reasons. No I disagree that they are reasons at all. Half of the ones you enumerated are nonsensical or irrelevant, the other half can be fixed by adding features to or fixing PyPI. One or two read like reasons why someone might actually decide not to upload something to PyPI but which that reason isn’t really all that reasonable and finally a grand total of one or two of them look like an actual legit reasons and it only applies to Crypto software. I mean your reasoning included things like: PyPI doesn’t let you upload two files with the same name, you gave the example of UC2 vs UC4 The thing being if PyPI did allow you to do that then we’d have no way to determine which one is the right one and half of the people would just get randomly failing installs because we guessed the wrong one. They don’t know it’s possible to upload to PyPI I don’t even know how to response to this one. PyPI used to be unreliable but isn’t now and they don’t know about it Nor this one. Not wanting people to download a package at all and instead check out from a VCS repo This’ll randomly fail for people depending on if they happen to have that VCS installed. Also I don’t see any evidence that people are actually doing this outside of supporting a foo==dev install which pip won’t install by default either. Not wanting to add latency for yourself for downloading from PyPI Instead you’re going to inflict extra latency on everyone else when you could just run a mirror or upload the packages to your own server. File types that PyPI doesn’t support and which utilize a third party package format
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On May 9, 2014, at 12:11 PM, Stefan Krah ste...@bytereef.org wrote: Donald, I'm out of his discussion. I have one last request: please don't gossip about core devs in public as long as you have commit privs: https://botbot.me/freenode/python-requests/ I don’t really know how to respond to this. I was talking to some people I know and I gave them a summary of what was happening. I stand by everything I said there and I don’t think any of it was wrong or even gossip at all. If people feel that was inappropriate then go ahead and take my commit privs. I have them to make it easier for me to write PEPs and to update ensurepip. If they’re going to be used as an excuse to attempt to censor me then I’d rather not have them as I generally always speak my mind and I won’t stop doing so. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On May 9, 2014, at 1:28 PM, R. David Murray rdmur...@bitdance.com wrote: On Fri, 09 May 2014 11:39:02 -0400, Donald Stufft don...@stufft.io wrote: On May 9, 2014, at 9:58 AM, M.-A. Lemburg m...@egenix.com wrote: On 09.05.2014 13:44, Donald Stufft wrote: On May 9, 2014, at 4:12 AM, M.-A. Lemburg m...@egenix.com wrote: I snipped the rest of the discussion and reliability, using unmaintained packages and projects using their own mirrors (which should really be the standard, not an exceptional case), because it's not really leading anywhere: Using your own mirror shouldn’t be the standard if all you’re doing is automatically updating that mirror. It’s a hack to get around unreliability and it should be seen of as a sign of a failure to provide a service that people can rely on and that’s how I see it. People depend on this service and it’s irresponsible to not treat it as a critical piece of infrastructure. I don't understand this. Why it is our responsibility to provide a free service for a large project to repeatedly download a set of files they need? Why does it not make more sense for them to download them once, and only update their local copies when they change? That's almost completely orthogonal to making the service we do provide reliable. Well here’s the thing right. The large projects repeatedly downloading the same set of files is a canary. If any particular project goes uninstallable on PyPI (or if PyPI itself goes down) then nobody can install it, the people installing things over and over every day or the people who just happened to be installing it during that downtime. However intermittent failures and general insatiability is going to be noticed by the projects who install things over and over again quicker and easier and thus it becomes a lot easier to use them as a general gauge for what the average “uptime” is. IOW if PyPI goes unavailable for 10 minutes 5 times a day, you might get a handful of “small” installers (e.g. not the big projects) in each downtime, but a different set who are likely to shrug it off and just call treat it as the norm even though it’s very disruptive to what they’re doing. However the big project is highly likely to hit every single one of those downtimes and be able to say “wow PyPI is flaky as hell”. To expand further on that if we assume that we want ``pip install foo`` to be reliable and not work sometimes and work at other times then we’re aiming for as high as uptime as possible. PyPI gets enough traffic that any single large project isn’t a noticeable drop in our bucket and due to the way our caching works it actually helps us to be faster and more reliable to have people constantly hitting packages because they’ll be in cache and able to be served without hitting the Origin servers. Just for the record, PyPI gets roughly 350 req/s basically 24/7, in the month of April we served 71.4TB of data with 877.4 million requests of which 80.5% never made it to the actual servers that run PyPI and were served directly out of the geo distributed CDN that sits in front of PyPI. We are vastly better positioned to maintain a reliable infrastructure than ask that every large project that uses Python to do the same. The reason that it’s our responsibility for providing it is because we chose to provide it. There isn’t a moral imperative to run PyPI, but running PyPI badly seems like a crummy thing to do. For perspective, Gentoo requests that people only do an emerge sync at most once a day, and if they have multiple machines to update, that they only do one pull, and they update the rest of their infrastructure from their local copy. To be clear, there are other reasons to run a local mirror but I don’t think that it’s reasonable to expect anyone who wants a reliable install using pip to stand up their own infrastructure. Further to this point here I’m currently working on adding caching by default for pip so that we minimize how often different people hit PyPI and we do it automatically and in a way that doesn’t generally require people to think about it and that also doesn’t require them to stand up their own infra. As another point of information for comparison, Gentoo downloads files from wherever they are hosted first, and only if that fails falls back to a Gentoo provided mirror (if I remember correctly...I think the Gentoo mirror copy doesn't always exist?). --David ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On May 9, 2014, at 4:20 PM, Terry Reedy tjre...@udel.edu wrote: On 5/9/2014 2:12 PM, Donald Stufft wrote: On May 9, 2014, at 1:28 PM, R. David Murray rdmur...@bitdance.com wrote: I don't understand this. Why it is our responsibility to provide a free service for a large project to repeatedly download a set of files they need? Why does it not make more sense for them to download them once, and only update their local copies when they change? That's almost completely orthogonal to making the service we do provide reliable. Well here’s the thing right. The large projects repeatedly downloading the same set of files is a canary. If any particular project goes uninstallable on PyPI (or if PyPI itself goes down) then nobody can install it, the people installing things over and over every day or the people who just happened to be installing it during that downtime. However intermittent failures and general insatiability is going to be noticed by the projects who install things over and over again quicker and easier and thus it becomes a lot easier to use them as a general gauge for what the average “uptime” is. I have had the same question as David, so I also appreciate your answer. IOW if PyPI goes unavailable for 10 minutes 5 times a day, you might get a handful of “small” installers (e.g. not the big projects) in each downtime, but a different set who are likely to shrug it off and just call treat it as the norm even though it’s very disruptive to what they’re doing. However the big project is highly likely to hit every single one of those downtimes and be able to say “wow PyPI is flaky as hell”. To expand further on that if we assume that we want ``pip install foo`` to be reliable and not work sometimes and work at other times then we’re aiming for as high as uptime as possible. PyPI gets enough traffic that any single large project isn’t a noticeable drop in our bucket and due to the way our caching works it actually helps us to be faster and more reliable to have people constantly hitting packages because they’ll be in cache and able to be served without hitting the Origin servers. Just for the record, PyPI gets roughly 350 req/s basically 24/7, in the month of April we served 71.4TB of data with 877.4 million requests of which 80.5% never made it to the actual servers that run PyPI and were served directly out of the geo distributed CDN that sits in front of PyPI. We are vastly better positioned to maintain a reliable infrastructure than ask that every large project that uses Python to do the same. The reason that it’s our responsibility for providing it is because we chose to provide it. There isn’t a moral imperative to run PyPI, but running PyPI badly seems like a crummy thing to do. Agreed. For perspective, Gentoo requests that people only do an emerge sync at most once a day, and if they have multiple machines to update, that they only do one pull, and they update the rest of their infrastructure from their local copy. To be clear, there are other reasons to run a local mirror but I don’t think that it’s reasonable to expect anyone who wants a reliable install using pip to stand up their own infrastructure. Ok, you are not saying that caching is bad, but that having everyone reinvent caching, and possibly doing it badly, or at least not in thebest way, is bad. Yea, caching isn’t in general a bad thing, and actually PyPI uses it heavily. All access to /simple/ and /packages/ is cached for 24 hours by our CDN unless someone uploads or deletes a file, in which case we selectively purge those URLs from the CDN cache so that they (nearly) instantly get the updated results. Warehouse (aka PyPI 2.0) is designed to utilize our CDN cache even further and I’m hoping to get our cache rate even higher using it. Further to this point here I’m currently working on adding caching by default for pip so that we minimize how often different people hit PyPI and we do it automatically and in a way that doesn’t generally require people to think about it and that also doesn’t require them to stand up their own infra. This seems like the right solution. It would sort of make each machine a micro-CDN node. Yes, it’s bog standard HTTP stuff just like a browser does it. The major difference is we’re limiting the maximum lifetime of a cache item in the client (pip) for the index pages but we are not doing that for the package files themselves. This is done to prevent a misconfigured server from causing pip to not see new versions for hours/days/weeks/years/whatever. Additionally this change also includes making pip smarter about HTTP requests in that if we have a stale item in the cache which as a Last-Modified or an ETag header we’ll do a conditional GET which will hopefully be returned with an HTTP 304 Not Modified so that we can simply refresh the stale item in the cache and use it again instead of needing
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
.) Whether a package is internal or external is orthogonal to both points. Again as I said above, the internal vs external isn’t a security related decision it is a “uptime” related decision. Verifiable vs unverifiable *is* a security related decision. With all these points, I find it questionable for an official install tool to make security related remarks about just one category of weaknesses. After all, people might be led to believe that pip is some sort of apt-get and all uploaded packages are safe. Stefan Krah [1] Note that the joke is quite innocent in comparison to what I've read on distutils-sig about the subject. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On May 8, 2014, at 9:39 AM, M.-A. Lemburg m...@egenix.com wrote: Well, to be fair and leaving aside uptime concerns and the general desire to always install packages from some server instead of a safe and trusted local directory (probably too obvious ;-), it would certainly be possible to add support for trusted externally hosted packages. There is support for trusted externally hosted packages, you put the URL in PyPI and include a hash in the fragment like so: http://www.bytereef.org/software/mpdecimal/releases/cdecimal-2.3.tar.gz#md5=655f9fd72f7a21688f903900ebea6f56 The hash can be md5 or any of the sha-2 family. Now this does not mean that ``pip install cdecimal`` will automatically install this, because whether or not you're willing to install from servers other than PyPI[1] is a policy decision for the end user of pip. The only real contention point there is whether installing from other servers should be on or off by default. PEP438 selected off by default, and I agree with that decision. Installing externally hosted files, which are able to be safely downloaded[2], was a surprising behavior to *everyone* I've talked to who hadn't already discovered that pip/easy_install did that. For the people it wasn't surprising too, they said it was surprising when they had originally discovered it[3]. [1] To be specific, other than the configured index(es), which happens to default to PyPI. [2] For the definition of safe that PyPI/pip operate under, which is that the author of a package is assumed to be trusted by the person electing to download their package. [3] I suspect people who were around when PyPI *couldn't* host files and were only an index would be the exception to this. However, for some reason there's a strong resistance against doing this, which I frankly don't understand. I agree with Stefan that the warning message wording is less than ideal. You'd normally call such blanket statements FUD, esp. since there are plenty external hosting services which are reliable and safe to use. I don't think the warning is FUD, and it doesn't mention anything security related at all. The exact text of the warning is in the subject of the email here: cdecimal an externally hosted file and may be unreliable Which is true as far as I can tell, it is externally hosted, and it may be unreliable[1]. If there is a better wording for that I’m happy to have it and will gladly commit it myself to pip. [1] In my experience dealing with complaints of pip's users, one of their big ones was that some dependency they use was, typically unknown to them, hosted externally and they found out it was hosted externally because the server it was hosted on went down. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On May 8, 2014, at 9:58 AM, Donald Stufft don...@stufft.io wrote: Now this does not mean that ``pip install cdecimal`` will automatically install this, because whether or not you're willing to install from servers other than PyPI[1] is a policy decision for the end user of pip. I forgot to add, for externally hosted files that are able to be safely downloaded pip's users can either elect to use any/all of them via: $ pip install --allow-all-external cdecimal $ PIP_ALLOW_ALL_EXTERNAL=1 pip install cdecimal $ echo --allow-all-external\ncdecimal requirements.txt $ pip install -r requirements.txt $ echo [global]\nallow-allow-external=true ~/.pip/pip.conf $ pip install cdecimal They can also elect to allow externally hosted files for *only* cdecimal using: $ pip install --allow-external cdecimal decimal $ PIP_ALLOW_EXTERNAL=cdecimal pip install cdecimal $ echo --allow-external decimal\ncdecimal requirements.txt $ pip install -r requirements.txt $ echo [global]\nallow-external=decimal ~/.pip/pip.conf $ pip install cdecimal I may have the syntax on the non command flag options slightly wrong, those are auto generated for us in our option system and I don't personally use those options. Also the reason for the extra verbosity in ``--allow-external`` is so you can allow it on something you don't directly depend on, like: $ pip install --allow-external cdecimal foobar-which-depends-on-cdecimal - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On May 8, 2014, at 10:11 AM, R. David Murray rdmur...@bitdance.com wrote: On Thu, 08 May 2014 09:58:08 -0400, Donald Stufft don...@stufft.io wrote: I don't think the warning is FUD, and it doesn't mention anything security related at all. The exact text of the warning is in the subject of the email here: cdecimal an externally hosted file and may be unreliable Which is true as far as I can tell, it is externally hosted, and it may be unreliable[1]. If there is a better wording for that I’m happy to have it and will gladly commit it myself to pip. [1] In my experience dealing with complaints of pip's users, one of their big ones was that some dependency they use was, typically unknown to them, hosted externally and they found out it was hosted externally because the server it was hosted on went down. unreliable reads as not safe, ie: insecure. You probably want something like and access to it may be unreliable. --David Done: https://github.com/pypa/pip/commit/69bf7067 - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On May 8, 2014, at 10:21 AM, R. David Murray rdmur...@bitdance.com wrote: On Thu, 08 May 2014 10:11:39 -0400, R. David Murray rdmur...@bitdance.com wrote: On Thu, 08 May 2014 09:58:08 -0400, Donald Stufft don...@stufft.io wrote: I don't think the warning is FUD, and it doesn't mention anything security related at all. The exact text of the warning is in the subject of the email here: cdecimal an externally hosted file and may be unreliable Which is true as far as I can tell, it is externally hosted, and it may be unreliable[1]. If there is a better wording for that Iâm happy to have it and will gladly commit it myself to pip. [1] In my experience dealing with complaints of pip's users, one of their big ones was that some dependency they use was, typically unknown to them, hosted externally and they found out it was hosted externally because the server it was hosted on went down. unreliable reads as not safe, ie: insecure. You probably want something like and access to it may be unreliable. Actually, thinking about this some more, *most* end-users aren't going to care that there's another point of failure here, they only care if it works or not when they try to install it. So something like cdecimal is not hosted on pypi; download may fail if external server is unavailable might be clearer. And once you're at that point, as a user I'm going to grumble, Well, why the heck didn't you just try?, as I figure out how to re-execute the command so that it does try. --David ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io Most users are not going to care up until the point where the external server is unavailable, and then they care a whole lot. On the tin it sounds reasonable to just download the external file if the server is up however we've done that for a long time and the end result has been end user pain. Now requiring someone to add a flag in order to download an externally hosted file is also end user pain. The difference between the two pains is when they happen. The requiring a flag pain happens at the point of decision, when you decide to make your deployment depend on something hosted externally. The default to allow pain happens sometime in the future, when you may or may not have any idea why suddenly your installs aren't working (and when you look, PyPI is up so you're really very confused why this particular file doesn't work). Even worse is the case when a project has some old files, but the newer versions aren't hosted and suddenly people are getting very old releases which is even more confusing to the end users. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On May 8, 2014, at 10:31 AM, Antoine Pitrou solip...@pitrou.net wrote: On Thu, 08 May 2014 10:21:34 -0400 R. David Murray rdmur...@bitdance.com wrote: unreliable reads as not safe, ie: insecure. You probably want something like and access to it may be unreliable. Actually, thinking about this some more, *most* end-users aren't going to care that there's another point of failure here, they only care if it works or not when they try to install it. So something like cdecimal is not hosted on pypi; download may fail if external server is unavailable might be clearer. And once you're at that point, as a user I'm going to grumble, Well, why the heck didn't you just try?, as I figure out how to re-execute the command so that it does try. Agreed. That warning looks rather pointless and only aimed at trying to enforce the pip developers' ideological preferences. Regards Antoine. The pip developers didn’t make this decision. It was discussed on distutils-sig hammered out in a PEP, and then accepted. We took part in that discussion, but ultimately we implemented PEP438. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On May 8, 2014, at 10:36 AM, Stefan Krah ste...@bytereef.org wrote: Donald Stufft don...@stufft.io wrote: There is support for trusted externally hosted packages, you put the URL in PyPI and include a hash in the fragment like so: http://www.bytereef.org/software/mpdecimal/releases/cdecimal-2.3.tar.gz#md5=655f9fd72f7a21688f903900ebea6f56 That is exactly the mode I was using until today. This mode produced the subject's warning message. Today I've switched to manual install mode with manual sha256sum verification which is *far* safer than anything you get via pip right now. It is not safer in any meaingful way. If someone is in a position to compromise the integrity of PyPI's TLS, they can replace the hash on that page with something else. Now you've attempted to work around this by telling people to go look up the release announcement hash. However if someone can compromise the integrity of PyPI's TLS, they can also compromise the integrity of https://mail.python.org/, or GMane, or any other TLS based website[1]. All of that assumes that the end user is going to bother to verify the hash *at all* which almost none of them will and they'll just check the http url into their requirements.txt file and be downloading things over HTTP and be vulnerable to arbitrary code execution via MITM. [2] For the definition of safe that PyPI/pip operate under, which is that the author of a package is assumed to be trusted by the person electing to download their package. No, there are other holes, which you have conceded in your previous mail. The presence of other holes is not a useful argument to avoid closing a hole. We're working to close all of them, and that ends up meaning we close one at a time. I don't think the warning is FUD, and it doesn't mention anything security related at all. The exact text of the warning is in the subject of the email here: cdecimal an externally hosted file and may be unreliable Which is true as far as I can tell, it is externally hosted, and it may be unreliable[1]. If there is a better wording for that I?m happy to have it and will gladly commit it myself to pip. Do you honestly not see a difference between the cited warning and the *intended* warning the server's availability may be unreliable”? Do I? No I don’t. However I’ve since adjusted the message based on R David Murray’s feedback to make sure it specifically says that access may be unreliable instead of just that the package itself may be unreliable. Even the latter is FUD or a truism (it applies to any server). No, because the use of an external host *adds* additional unreliability. If PyPI is down, then all packages are down, including ones that host externally. If the cdecimal server is down, then that one specific package is unavailable. It is impossible to do anything but reduce the overall availability by adding additional SPOFs. The real question is: Why is there a warning if the person running pip has explicitly allowed external packages? Why is there a warning? Originally that warning was there because the first part of the transition to this mode of defaults was to give an option to disable externally hosted files, but leave it on by default. In this phase we gave this warning to tell the people who just leave things to their default about this file. Should the warning itself still exist? I don't know, but a better avenue for asking for a change in pip is via our issue tracker instead of whining on python-dev. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On May 8, 2014, at 11:19 AM, Stefan Krah ste...@bytereef.org wrote: Donald Stufft don...@stufft.io wrote: hosted packages are brittle and more prone to failure. Every single external server adds *another* SPOF into any particular install set. Even if every external server has a 99.9% uptime, when you combine multiple of them the total uptime of any particular set of requirements drops quickly. This has been a major complaint that people have had over time. We have been through that many times; to me this is an indication that people are using pip under circumstances when they should not. pip is not apt-get. [I am aware that *you* know that, just stating it again for the broader audience.] 2) Last time I looked, access credentials (via lost login) were sent out in plaintext. The existence of other security issues is not an excuse to not fix a security issue. There are other problems and we're slowly working on trying to clear them out. It is, however, a reason to avoid error messages that could *imply* (rightly or wrongly) to users that the combination of pip and internal packages is safe. 3) AFAIK people can upload a different (malicious) version of a package with *the exact same name*. Yes, a malicious author is needfully outside of the threat model for PyPI/pip. How so? I'm avoiding this attack by publishing sha256sums at release time. The point is that I *cannot* change cdecimal-2.3.tar.gz without a user digging up a checksum mismatch from the mailing list archives. Practically speaking exactly zero people will ever bother to dig up the checksum from a mailing list archive, or even verify the hash that you have right on PyPI itself. Worse security that people actually use is infinitely better than better security that nobody uses. 6) D.J. Bernstein, who is somewhat security minded, has been shipping his software *for years* with just plain HTTP and published checksums. Argument from authority doesn't really hold up very well when DJB doesn't distribute is software in a way that is intended to be consumed mechanically. Also while he may be a crypto expert he is far from an expert on successfully distributing software, unless you also think that the signature checking in most OS provided package managers is pointless. That is sort of a strawman. The whole point *is* that certain distribution models don't lend themselves to mechanical consumption. I cannot speak for DJB, perhaps he is just thinking that GPG signing is pointless if users can't validate the signature and SSL is pointless because one cannot trust governments. OS package signing is useful since the packages are curated. If anyone could upload a package to Debian, whereupon it would be signed with the official key, apt-get would lose its usefulness. People are going to mechanically consume them. You can tell them it’s wrong all you want but they are going to do it. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On May 8, 2014, at 11:21 AM, R. David Murray rdmur...@bitdance.com wrote: On Thu, 08 May 2014 10:37:15 -0400, Donald Stufft don...@stufft.io wrote: Most users are not going to care up until the point where the external server is unavailable, and then they care a whole lot. On the tin it sounds reasonable to just download the external file if the server is up however we've done that for a long time and the end result has been end user pain. Now requiring someone to add a flag in order to download an externally hosted file is also end user pain. The difference between the two pains is when they happen. The requiring a flag pain happens at the point of decision, when you decide to make your deployment depend on something hosted externally. The default to allow pain happens sometime in the future, when you may or may not have any idea why suddenly your installs aren't working (and when you look, PyPI is up so you're really very confused why this particular file doesn't work). Even worse is the case when a project has some old files, but the newer versions aren't hosted and suddenly people are getting very old releases which is even more confusing to the end users. Ah, I understand now. Your perspective is as someone who is using pip for *deployment*. Deployment, or any kind of situation where you want to have a reproducible build. Generally via deployment yes. I'm speaking from a python+plus+pip end-user perspective, which is going to be even more common now that it is part of the Python distribution. I'm not sure how you reconcile these two worlds. I would venture to suggest that if you are using it for deployment you really ought to be using a local package repository[*], not the global one; but, as someone observed, the sensible thing to do and what people actually do are often very different; and, since I haven't done this particular kind of deployment scenario in Python myself, there may be reasons I'm missing. People simply don’t do this[1]. Especially in a world with things like Heroku existing which makes it stupid simple to use pip to install from PyPI but installing from your own server requires standing up some infrastructure. However, your last mention of end users confuses me. Why are end users getting old packages in a deployment situation? Isn't it the developer/operations people (and the latter would, I assume, control the 'external packages' flag) who would be seeing that? Maybe you mean something by deployment different from how I use the word? Someone using pip, this may be a developer who is just trying to recreate their production environment locally, this may be someone using chef/puppet to automate installing via pip, this may be someone pushing to Heroku. The old versions thing is more that it’s really confusing when you type ``pip install foo`` on a monday and get 2.0, and ``pip install foo`` on weds and get 0.4. --David [*] I found it *such* a pain to do this for perl/cpan. I have a project for a customer where I have to do this, and the hoops I had to jump through to get a reliable deployment (where packages wouldn't be unexpectedly upgraded under my feet) were nasty. (This was several years ago and I haven't revisited it, so maybe things have gotten better, or I just missed something.) I haven't had to do it for python yet, oddly enough, so I don't know how hard it is for Python. For Python with pip you can use a requirements.txt file to create a set of dependencies that are pinned to exact versions like: foo==2.0 bar==2.3 And pip will (theoretically, our dep solving is real bad ATM) install exactly those versions from your index server. Generally this means PyPI which means the author can delete the version and upload a new file with the same version number. However it’s also trivial to stand up your own server. It can be as easy as pointing nginx/Apache at a static directory with autoindex = True. (See: https://wheels.caremad.io/). On top of that there is peep which adds a secure message digest on it to make sure that the author/index didn’t swap things out on you, and there is some discussion about how best to add that to pip itself. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On May 8, 2014, at 11:34 AM, Stefan Krah ste...@bytereef.org wrote: Donald Stufft don...@stufft.io wrote: Today I've switched to manual install mode with manual sha256sum verification which is *far* safer than anything you get via pip right now. It is not safer in any meaingful way. If someone is in a position to compromise the integrity of PyPI's TLS, they can replace the hash on that page with something else. Now you've attempted to work around this by telling people to go look up the release announcement hash. However if someone can compromise the integrity of PyPI's TLS, they can also compromise the integrity of https://mail.python.org/, or GMane, or any other TLS based website[1]. Of course it is safer. Suppose a file is stored on PyPI: 1) Attacker guesses my username (or is it even visible, I'm not sure). 2) Clicks on lost login. 3) Intercepts mail (difficult, but far from the TLS attack category). Maybe on a home or university network. Or a rogue person at a mail provider. 4) Changes the uploaded file together with the hash. pip would be perfectly happy, checking the hash via Google would turn up a mismatch. I said “meaningful”. Almost nobody is going to ever bother googling it and the likelihood that someone is able to MITM *you* specifically is far lesser than the likelihood that someone is going to MITM one of the cdecimal users. Additionally your messages aren’t signed and email isn’t an authenticated profile so if someone was able to get your password they could simply spoof and email from you to the mailing list with new hashes, or edit out the description telling people to go google some stuff. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On May 8, 2014, at 11:37 AM, M.-A. Lemburg m...@egenix.com wrote: On 08.05.2014 16:42, M.-A. Lemburg wrote: On 08.05.2014 15:58, Donald Stufft wrote: On May 8, 2014, at 9:39 AM, M.-A. Lemburg m...@egenix.com wrote: Well, to be fair and leaving aside uptime concerns and the general desire to always install packages from some server instead of a safe and trusted local directory (probably too obvious ;-), it would certainly be possible to add support for trusted externally hosted packages. There is support for trusted externally hosted packages, you put the URL in PyPI and include a hash in the fragment like so: http://www.bytereef.org/software/mpdecimal/releases/cdecimal-2.3.tar.gz#md5=655f9fd72f7a21688f903900ebea6f56 The hash can be md5 or any of the sha-2 family. Now this does not mean that ``pip install cdecimal`` will automatically install this, because whether or not you're willing to install from servers other than PyPI[1] is a policy decision for the end user of pip. Hmm, if you call that feature trusted externally hosted packages, pip should really do trust them, right ? ;-) I can understand that pip defaults to not trusting URLs which don't meet the above feature requirements, but not that it still warns about unreliable externally hosted packages even if the above feature is used. At the moment, pip will refuse to use an externally hosted files even if the package author uses the above hashed URLs; even with HTTPS and proper SSL certificate chain. Could this perhaps be changed/reconsidered for pip ? Note that easy_install/setuptools does not have such problems. Anything can be changes or reconsidered of course. I feel pretty strongly that an installer should not install things from places other than the index without a specific opt in. That discussion would be best done on distutils-sig as it would require reversing the decision in PEP438. I really don't feel strongly one way or the other about the *warning* that happens when you allow an external file. It exists primarily because at the time it was implemented external files were default to allowed. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On May 8, 2014, at 12:03 PM, Stefan Krah ste...@bytereef.org wrote: Donald Stufft don...@stufft.io wrote: I said ?meaningful?. Almost nobody is going to ever bother googling it and the likelihood that someone is able to MITM *you* specifically is far lesser than the likelihood that someone is going to MITM one of the cdecimal users. I'm doing this for important installs. -- That is how I installed qmail and djbdns. Additionally your messages aren?t signed and email isn?t an authenticated profile so if someone was able to get your password they could simply spoof and email from you to the mailing list with new hashes, or edit out the description telling people to go google some stuff. Signing messages is pointless if the key isn't well connected. Also, I'm reading the lists and would notice a release. Most importantly, the checksum mismatch would still be found, since the old messages with the correct sum would still exist under the scenario we're talking about (i.e. not GHCQ hacking into Belgacom routers). I’m unsure if you’re being willfully dense or if you’re just not understanding what I mean when I say “almost”. Of course there are going to be a few outliers where people do bother to do that, but it’s not going to be common place at all. But whatever, I’ve removed the warning that occurs when you install an externally hosted file [1] and it will be included in pip 1.6. I have not changed the defaults for --allow-all-external nor have I removed the warning that occurs when someone elects to install an unverifiable download. [1] https://github.com/pypa/pip/commit/9f56b79e8d - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On May 8, 2014, at 12:42 PM, R. David Murray rdmur...@bitdance.com wrote: On Thu, 08 May 2014 11:32:28 -0400, Donald Stufft don...@stufft.io wrote: On May 8, 2014, at 11:21 AM, R. David Murray rdmur...@bitdance.com wrote: Ah, I understand now. Your perspective is as someone who is using pip for *deployment*. Deployment, or any kind of situation where you want to have a reproducible build. Generally via deployment yes. [...] For Python with pip you can use a requirements.txt file to create a set of dependencies that are pinned to exact versions like: foo==2.0 bar==2.3 And pip will (theoretically, our dep solving is real bad ATM) install exactly those versions from your index server. Generally this means PyPI which OK, this makes sense, then. (I wish perl/cpan had something similar...maybe it does, but I couldn't find it at the time.) This still leaves the fact that there is a disconnect between the needs of two different audiences for PIP: people who deploy things, and everyone else who just uses pip to install stuff. Yup balancing between the two is something we have to do in every decision we make. When PEP438 was being discussed I did a pretty extensive amount of investigation into what affect this change would have [1]. What I found was that: - The sizable majority was projects would host things on PyPI - There was a significant chunk of projects where a single release or two would only be available externally and it was an accident that they weren’t uploaded. - Of the links that were available externally, very few of them were available in a way that was verifiable and were thus insecure to install. Because of this it was determined that simply allowing externally hosted files without also allowing externally hosted and unverified files would not actually have a significant impact for the vast bulk of the projects that were not hosted on PyPI. The second group is going to overwhelm the first group, if it doesn't already. Generally yes, because not every who uses pip to deploy uses pip to install locally, but most people who use pip to deploy also use pip locally. And I think that's all the comments I have on this issue :) [1]: https://github.com/dstufft/pypi.linkcheck - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On May 8, 2014, at 5:02 PM, Paul Moore p.f.mo...@gmail.com wrote: On 8 May 2014 16:46, Donald Stufft don...@stufft.io wrote: Anything can be changes or reconsidered of course. I feel pretty strongly that an installer should not install things from places other than the index without a specific opt in. That discussion would be best done on distutils-sig as it would require reversing the decision in PEP438. I think it's worth reconsidering. Since this behaviour was implemented, there have been many instances of confusion and unhappiness with the situation, both from package developers and pip users. I don't think that's good for pip. I would like to see PEP 438 reviewed with the intention of working out how to fix the user experience (ideally while retaining the reliability enhancements, but accepting that compromises may be needed). I think most of the confusion has been over the fact that —allow-external takes a package name, not that it exists at all. Some points: 1. Often a user won't know that a package is externally hosted until an install fails. Having to add a flag and retry is a lousy experience. Why not at a minimum have an externally hosted - are you sure? prompt? A prompt is OK with me. 2. The way the flags are implemented (notably the need to repeat the package name) is clearly confusing and irritating for users, even if the reasons are logical. We should look at fixing that. Yea, there’s a ticket for this. 3. The user complaints against pip are significant and ongoing. I don't think they should be ignored. If PEP 438 cannot be reconsidered in the light of user feedback, then I think the pip developers may need to have a discussion about whether we conform to the PEP (clearly Donald thinks we should, I'm not sure I do, and I don't know about the others). But I don't think it's healthy for pip to be looking at deliberately ignoring an accepted PEP, so I'd prefer it if the debate was around addressing the issues in the PEP itself. I don’t think the problem with with the PEP. 4. Maybe distutils-sig *isn't* the right place to discuss revising PEP 438. The issues getting raised are end-user problems, and the users most affected are unlikely to be on that list. Socially, this change does not seem to be having the effect of persuading more package developers to host on PyPI. The stick doesn't appear to have worked, maybe we should be trying to find a carrot? Do you have any data to point to that says it hasn’t worked? Just to see what impact it has had, I’m running my scripts again that I ran a year ago to see what has changed, already I can see they are processing MUCH faster than last year. Or maybe we have to accept that some developers have sound reasons for not hosting on PyPI and work with them to find an acceptable compromise? Has anyone checked what Stefan's reasons are for not hosting cdecimal on PyPI? Do they represent a use case that the PEP hasn't considered? If I recall correctly his reasoning is that he finds the legal requirements associated with uploading to PyPI to be unsatisfactory. I really don't feel strongly one way or the other about the *warning* that happens when you allow an external file. It exists primarily because at the time it was implemented external files were default to allowed. I think it's reasonable to remove the warning. If the user chooses to allow an external file, it makes sense to assume they understand the implications and not nag them about their decision. Particularly given the level of controversy the warning is generating. The warning is gone as of a few hours ago. On a personal note, I'm uncomfortable with the way this change is perceived as a case of *pip* enforcing a behaviour that the pip developers feel should be required. I actually don't like this change particularly. So having pip implement the behaviour required by that PEP is to me simply a case of compliance with the agreed standard. But now, as a pip developer, being held responsible for the resulting user pain, and being expected to defend it, does not make me happy. Paul I think the pain is being overrepresented and the positives are being ignored. The problem is the benefits of this PEP are much like the benefits of TLS too. For the vast majority of people they don’t notice anything different except installing things is faster and more reliable. They don’t attribute that to the PEP or this decision, they just internalize it as the new norm. However the people who this does affect will seek out why it broke and raise an issue citing that thing specifically. This creates a perception of lots of pain for no gain when the reality is not that. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On May 8, 2014, at 6:20 PM, Nick Coghlan ncogh...@gmail.com wrote: On 9 May 2014 07:23, Donald Stufft don...@stufft.io wrote: On May 8, 2014, at 5:02 PM, Paul Moore p.f.mo...@gmail.com wrote: Or maybe we have to accept that some developers have sound reasons for not hosting on PyPI and work with them to find an acceptable compromise? Has anyone checked what Stefan's reasons are for not hosting cdecimal on PyPI? Do they represent a use case that the PEP hasn't considered? If I recall correctly his reasoning is that he finds the legal requirements associated with uploading to PyPI to be unsatisfactory. I actually need to follow up on that, because the terms *were* legally questionable last time I looked (also too hard to review, since as far as I am aware, they're only presented during new user sign-up). I'll deal with that at work today. I’m pretty sure VanL wrote the terms and has explicitly said they won’t change and are exactly as broad as they need to be without being any broader[1]. They are linked to from the footer of every UI centric PyPI page. [1] https://mail.python.org/pipermail/python-legal-sig/2013-March/03.html - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On May 8, 2014, at 5:22 PM, Donald Stufft don...@stufft.io wrote: Socially, this change does not seem to be having the effect of persuading more package developers to host on PyPI. The stick doesn't appear to have worked, maybe we should be trying to find a carrot? Do you have any data to point to that says it hasn’t worked? Just to see what impact it has had, I’m running my scripts again that I ran a year ago to see what has changed, already I can see they are processing MUCH faster than last year. The data has finished processing, it represents a time diff of approximately one year. The pip release that caused all of this was released about 4-5 months ago. Overall PyPI has seen a 50% growth in installable projects in that time. If the change would have had no effect we'd expect to see a ~50% increase across the board. However what we've seen is a a 60% (+10% of expected) increase in projects that can only be installed from PyPI and a 12% decrease in projects that have any unsafe files (-62% of expected). Further more we can see that if pip were to change the default of --allow-all-external it would take 23 projects from unable to be installed by default to able to be installed by default. This represents 0.2% of installable projects on PyPI. It would take an additional 40 projects and make one or more additional files able to be downloaded by default. Some other data points: * We've gone from 86% of projects being installable from PyPI to 92%. * We've gone from 5% of projects being only unsafely installable to 3% * We've gone from 14% of projects having any files unsafe to install to 8% * We've gone from 0.004% of projects being safely hosted externally to 0.2% Looking at these numbers I think it's safe to say that in this time period that the hosting hygiene of a PyPI project is more likely to be a better state than it was a year ago. We cannot state for a fact if this is because of this change or not, however given that the fallout is ~23 (or ~63) projects out of 38,835 I think it is incredibly reasonable to leave the defaults alone since there is a reasonably high chance that they played at least some part in that change. I'd love to get these numbers to the point where the number of projects installable strictly from PyPI is 100% (or at least 100% installable safely), however 92% (or 92.2%) is getting pretty close to that and hopefully that number will just continue to grow until it hits 100%. For reference, here's the raw numbers as well as some summary of the data here: https://gist.github.com/dstufft/b14008d11c0a5760dbed And the repository where the raw data as well as the scripts used to collect and process it is here: https://github.com/dstufft/pypi.linkcheck linkcollector.py collections while linkwriter.py writes out the json file, and stats2.py processes and gives the numbers from the gist above. links.json is the data from a year ago, and 2014-05-08.links.json is the data from today. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On May 9, 2014, at 12:34 AM, Donald Stufft don...@stufft.io wrote: The data has finished processing, it represents a time diff of approximately one year. The pip release that caused all of this was released about 4-5 months ago. Oh I forgot to mention: In order to make the comparison as accurate as possible I've used the same collection script as I did a year ago. This is prior to the real advent of Wheels so these numbers do not take into accounts Wheels at all (which pip can also install) but it *does* include Eggs which pip cannot install. Further more it also includes #egg=dev urls which point to a VCS (and it would consider those an unverifiable/unsafe download). - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pep8 reasoning
On Apr 25, 2014, at 5:52 PM, Carl Meyer c...@oddbird.net wrote: On 04/25/2014 01:55 PM, Ethan Furman wrote: On 04/25/2014 12:45 PM, Florent wrote: 2014-04-25 18:10 GMT+02:00 Nick Coghlan: And if you're going to publish a tool to enforce your *personal* style guide and include your own custom rules that the this is OK examples in PEP 8 fail to satisfy, don't call it pep8. Two cases where signaled in the issue #256 last month, where the tool is arguably not compliant with some of the current conventions of the PEP. I've highlighted the reasons behind these checkers in the issue tracker directly. I disagree that they are in direct opposition with the PEP 8 coding conventions, though. The problem is that you've named it pep8. To me, that means I can run it and get PEP 8 results. If I have to add a manual configuration to get actual PEP 8 semantics, it's a buggy tool. For a similar example, I am the author/maintainer of enum34, which purports to be the backport of Python's 3.4 enum class. While I could go and make changes to it to better match my style, or even the styles of other users, calling it enum34 after that would be misleading as some one couldn't then switch from using enum34 in Python 3.2 to using the default enum in Python 3.4. If you had extra switches to turn on extra behavior, that would be fine. Leaving it as it is, and calling it something else would be fine. But as it stands now, with the name of pep8 and the behavior of failing on the PEP 8 document... well, that's false advertising. I think this fuss is unreasonable and unwarranted. I'd like to thank Florent for taking the time to maintain an extremely-useful tool that makes it feasible to keep to a consistent PEP 8 style throughout a large codebase with many contributors, and I think he should have the leeway as maintainer to make the necessary judgment calls about precisely which PEP 8 recommendations are reported precisely how by the tool, given that: - we aren't talking about real variance from the spirit or recommendations of PEP 8 (though you wouldn't know it from some of the rhetoric here about personal preferences) - the tool makes it very easy to turn off whichever errors you don't prefer to enforce. - PEP 8 changes regularly (as Florent noted, the offending code example was only added recently), and there is value in the automated tool maintaining some stability for its users. Personally, I would much rather see pep8.py err on the side of being too strict with PEP 8's recommendations than too loose. Again, it's not hard to turn off the ones you don't want. If python-dev wants to control the precise behavior of pep8.py, bring it into the standard library and adopt maintenance of it. Otherwise, please give Florent some grace. Carl Carl’s post mirrors my own thoughts and it’s said much better than I could have. pep8.py doesn’t violate PEP8, it just takes a stricter view of it. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pep8 reasoning
On Apr 25, 2014, at 7:56 PM, Florent florent.xicl...@gmail.com wrote: 2014-04-26 0:46 GMT+02:00 Nick Coghlan ncogh...@gmail.com: Florent is claiming the endorsement of the PEP 8 authors and the consensus of python-dev for the tool's default behaviour (as noted above, this makes it personal for me, as I am a co-author of PEP 8). You're a co-author of PEP 8 since less than a year. I'm the maintainer of the pep8 tool since 2010. You should probably read the LICENSE file which is shipped with the pep8 too, and the disclaimer that I've posted previously. Never I engage the responsibility of the authors of the PEP 8 document, and I don't give any guarantee of being a *strict* PEP 8 compliance tool. I agree that I’ve never taken the name to mean that you’re claiming any sort of endorsement. There are a *vast* number of projects that implement something that was defined somewhere else and I don’t think any reasonable person can assume that all of those tools are endorsed by the authors of what they are implementing. However, you should notice that your ticket in the tracker is opened for 2 months only, and I did not flagged it as being resolved. As I've stated in my previous mail, I give priority to bugs over other requests. And even if you think it is a critical bug for yourself, it did not appear like that for the thousands of people which used the library for the last few years. But if you read the documentation carefully, you can see that I've already excluded some checks from the default behavior in the previous releases, when there was an argument which was backed by the PEP 8 itself: In the default configuration, the checks E123, E133, E226, E241 and E242 are ignored because they are not rules unanimously accepted, and PEP 8 does not enforce them. http://pep8.readthedocs.org/en/latest/intro.html#error-codes The question stay opened for issue #256 and codes E121 and E701. As I said before, they are not against PEP 8, they interpret some words. If you're nitpicking, you can probably reject half the checks of the pep8 tool with similar allegations. I *want* to be able to recommend this tool universally. But at the moment, I cannot, as its name is a lie: it enforces rules I don't personally agree with. Maybe I prefer you don't recommend it in the PEP 8 documentation if it means I'll be tied to your personal preferences and that I'll be forced to patch it every now and then when you change a line, or a punctuation in a PEP 8 example. At the end, I find you're a bit rude when you come to this project which is not endorsed by the PSF or any Python-Dev related entity and you say this project is wrong, because the developer did not obey to me when I order to remove that feature. You're more sympathetic and less in a hurry when it comes to some languishing bug on b.p.o :-) If you're so impatient, let's fork it and put in in cpython/Tools/ I would not fight against it. Sincerely, -- Florent ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pep8 reasoning
On Apr 25, 2014, at 7:20 PM, Ethan Furman et...@stoneleaf.us wrote: On 04/25/2014 03:26 PM, Donald Stufft wrote: pep8.py doesn’t violate PEP8, it just takes a stricter view of it. If pep8 reports errors on things that PEP 8 says are okay, that's a violation. -- ~Ethan~ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io Not really, any code that passes the pep8.py check is perfectly valid in the eyes of PEP8, if a check was implemented to say, require camelCase method names, then that would be a violation of a check. Being stricter is not a violation, it’s being stricter. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pep8 reasoning
On Apr 26, 2014, at 12:31 AM, Stephen J. Turnbull step...@xemacs.org wrote: Florent writes: I wrote some words in the documentation, one year ago, to explain what is the purpose of the tool and its limitations. There's no claim of any endorsement implicit or explicit by the PSF, the PSU or any other python developer :-) Of course there is an implicit claim of endorsement: the name. Read the TeX license -- pretty much the only restriction is *you can't call it TeX or any variation including that trademark if it doesn't pass TRIP*. When Don Knuth cares that much that names denote conformance to standard, Nick is in really good company. No there isn’t. For instance there are things on PyPI named after websites, like github, twitter, Facebook, etc. These things are not written by the companies behind those websites and are merely written to interface with those websites. Should we assume that those companies all endorse every single one of those projects simply because they chose a descriptive name for their project? The name indicates what it attempts to do, create a PEP8 linter, it makes no claim of endorsement by the authors of the PEP. You can say it’s confusing if it’s named pep8 but doesn’t actually make it “PEP8”, but claiming it’s endorsement isn’t correct. I agree 100% with Nick: in a program named pep8, the examples in PEP 8 should *all* pass in the sense of not being labelled errors. Of course if the PEP changes that doesn't mean you should withdraw or rename the program, or even that you are required to address the issue within any time span. But you should consider it a bug. Or required to address it at all if you don’t wish to. Since the issue is still open I assume that means that the author hasn’t decided what the correct remediation is yet. That said, issuing *warnings* for discouraged-but-sometimes-allowed practices is a great idea. A stricter warnings-are-errors *mode* is a good idea, especially for automated checkers, but it should be configurable for different policies. http://pep8.readthedocs.org/en/latest/intro.html#disclaimer Disclaimers are for lawyers. Other people assume names mean what they say. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 469: Restoring the iterkeys/values/items() methods
On Apr 19, 2014, at 12:35 PM, Guido van Rossum gu...@python.org wrote: I am also concerned about the dependency on Python 3.5 that we're building here. I'd much rather be able to use Twisted sooner, with 3.3 or at least 3.4. Anyone who is planning on using the bytes modulo formatting is going to be 3.5+ anyways. It seems like trying to fit as many of these compatibility things as Python is willing to do into 3.5 is the best possible solution since it’s likely that for a lot of these hanger-ons 3.5 is likely to be a minimum target anyways. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Language Summit notes
On Apr 18, 2014, at 2:31 PM, Antoine Pitrou solip...@pitrou.net wrote: On Fri, 18 Apr 2014 12:04:10 + Kristján Valur Jónsson krist...@ccpgames.com wrote: 2. Feature enhancement to 2.8. Take a robust and popular version of python and add some of the language goodies that have been added to 3.x and that don’t have an inherent 3.x aspect. Yield from. New exception model. Stdlib enhancements such as futures. The argument goes like this: We have a very popular platform out there with lots of momentum. People want incremental enhancements to it. Why not give them what they want? Bread and games and all that? A Rockband cannot stay cooped up in a studio producing experimental concept albums all the time. That is death. Sometimes it needs to go on tour and play old hits for the fans! I don't think we have recent download numbers since the Website overhaul (do we?), but Python 3 isn't an experimental concept language anymore (it hasn't been since 3.3 or 3.2, I'd say). Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io Fastly logs are stored in Dreamhost so we could make those numbers if they aren’t already available. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Software integrators vs end users (was Re: Language Summit notes)
On Apr 18, 2014, at 3:18 PM, Nick Coghlan ncogh...@gmail.com wrote: At this point, however, I'm mainly looking for consensus that there *are* two different problems to be solved here, and solving them both well in a single tool is likely to be nigh on impossible. (I'm aware we're really on the wrong list for that discussion, but I also think there's value in getting some broader python-dev awareness of this particular topic) I’m not sure about this? I mean yes those are two different areas, but I’m not sure about the split between Conda and pip here. As far as I can tell Conda is useful in the same way apt-get or yum is useful, you get a non Python specific set of packages (because sometimes things aren’t pure python) and you also remove a little bit of thinking about versions (although honestly I think it’s possible to remove most of that for consumers of packages). To be quite frank, a lot of the benefit of Conda outside of the “I need something that isn’t strictly Python) is in the fact they can bootstrap compiled packages whereas pip/wheel/PyPI combination we need to convince authors to upload their own binary packages (or at least develop something to make it easier like a build farm). - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
