Re: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement?
unfortunately common scenario of: > >sudo easy_install blah # --> installs to system Python 2.7 >python -c 'import blah' # --> fails because using newer Python 2.7 > > This may very well be the most common current usability problem today > with python.org (and other third-party) Pythons on OS X. > > With this PEP, we have an opportunity to fix this problem in one of > two ways. As long as pip requires setuptools and we do not inhibit > the installation of the easy_install scripts is to the same directory > as the python command itself, the new easy_install scripts will > shadow the Apple-supplied ones just as the python commands do. If it > is decided to not install the easy_install scripts now or in the > future (e.g. when pip no longer depends on setuptools), > ensurepip could install dummy easy_install scripts (if there > are none already installed there) that merely emit a message to the user > to encourage migration to pip and suggest, if needed, to use pip to install > setuptools to replace the placeholder easy_install. Either approach > would be a huge usability improvement over the situation today. > > (The actual implementation details for either approach are slightly more > complicated for OS X framework builds like the python.org installers > as the actual scripts directory is a ``bin`` directory within the > framework itself with optional symlinks from ``/usr/local/bin`` to > the fw bin directory for specific commands. The installer will now need > to also install symlinks in ``/usr/local/bin`` for ``pip`` and, as > proposed above, for ``easy_install``.) So this is an interesting question. On one hand I would say we shouldn't be installing easy_install as that feels very user facing to me and the fact that setuptools is being installed is an implementation detail. On the other hand if we put in stub commands that just simply direct the user to use pip then people who *want* to use easy_install (perhaps for Egg support) won't be able too (unless perhaps something is released on PyPI that restores the easy_install commands?) > >> A common source of Python installations are through downstream distributors >> such as the various Linux Distributions [#ubuntu]_ [#debian]_ [#fedora]_, OSX >> package managers [#homebrew]_, or Python-specific tools [#conda]_. In order > > If you are going to call out Homebrew, you should include the other major > OS X package managers, MacPorts and Fink. > >> .. [#homebrew] `Homebrew <http://brew.sh/>` > > .. [#macports] `MacPorts <http://macports.org>` > .. [#fink] `Fink <http://finkproject.org>` I can add them, I added homebrew because it's what I use :) Didn't really think to hard about being super inclusive. > > One final comment is that the PEP does not go into any detail on how it > will be implemented. As it stands, there is a fair amount of one-time > work, including implementing ensurepip, changes to the Windows installer, > changes to the OS X installer, documentation changes, all with testing and > backporting to 2.7.x and 3.3.x. Then there are the on-going process changes > for all future releases, impacting all release team members, which will need > to be documented in PEP 101. Do we have an understanding of who is do the > big pieces and by when? For the record, I am willing to do the extra > one-time and ongoing work for the OS X installers but it would be helpful > to know how we are going to get it all done in time for upcoming releases. To further expand upon my original answer, I'm volunteering to do the initial work on the ensurepip library, the scripts that will automated the ongoing maintenance work, and the back porting of both of the above. I can also attempt to do the OSX Installer and Windows installer but I have zero idea how the installers actually function so it's probably better for someone else to do that. Since it sounds like you're willing to do the work for the OSX installer that saves me from having to flail around trying to figure out how to do that part, so hopefully MvL or someone can do the same with the Windows installer. I'm not sure what needs done outside of the up front work, do I just propose changes to PEP 101? Do I make a whole new PEP? Is there more than just updating PEP 101? > > -- > Ned Deily, > n...@acm.org > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/donald%40stufft.io - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement?
On Sep 23, 2013, at 7:34 PM, Ned Deily wrote: > One final comment is that the PEP does not go into any detail on how it > will be implemented. As it stands, there is a fair amount of one-time > work, including implementing ensurepip, changes to the Windows installer, > changes to the OS X installer, documentation changes, all with testing and > backporting to 2.7.x and 3.3.x. Then there are the on-going process changes > for all future releases, impacting all release team members, which will need > to be documented in PEP 101. Do we have an understanding of who is do the > big pieces and by when? For the record, I am willing to do the extra > one-time and ongoing work for the OS X installers but it would be helpful > to know how we are going to get it all done in time for upcoming releases. I'm in a meeting right now so I only skimmed this email, but I wanted to mention here that I've been assuming I would be writing the python portions of this. I don't know how to do the OSX/Window installer pieces and I was hoping that the relevant people would be at least willing to help me figure it out or would implement the parts that need to be done inside the installers (mostly running the command). - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 453 Round 4 - Explicit bootstrapping of pip in Python installations
I've updated the PEP to include the solution for Debian that Nick mentioned and to clarify the behavior in the upgrade case. Online: http://hg.python.org/peps/rev/6c1b658bc16c diff -r 2899ba3bcef1 pep-0453.txt --- a/pep-0453.txt Thu Sep 19 09:23:03 2013 -0400 +++ b/pep-0453.txt Thu Sep 19 10:12:59 2013 -0400 @@ -294,7 +294,9 @@ This ensures that, by default, installing or updating CPython will ensure Update PEP453 to address Debian's concerns and Antoine's that the installed version of pip is at least as recent as the one included -with that version of CPython. +with that version of CPython. If a newer version of pip has already been +installed then ``python -m extractpip --upgrade`` will simply return without +doing anything. Installing from source @@ -532,6 +534,12 @@ * This may take the form of separate packages with dependencies on each other so that installing the Python package installs the pip package and installing the pip package installs the Python package. + * Another reasonable way to implement this is to package pip separately but +ensure that there is some sort of global hook that will recommend +installing the separate pip package when a user executes ``pip`` without +it being installed. Systems that choose this option should ensure that +the ``pyvenv`` command still installs pip into the virtual environment +by default. * Do not remove the bundled copy of pip. --------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 453 Round 4 - Explicit bootstrapping of pip in Python installations
On Sep 19, 2013, at 9:50 AM, Paul Tagliamonte wrote: > On Thu, Sep 19, 2013 at 11:48:49PM +1000, Nick Coghlan wrote: >> That's OK, especially if running "pip" recommends installing "python-pip" and >> "pyvenv" and "extractpip" are also separated out so that running "pyvenv" >> recommends installing a suitable package that depends on both Python and >> "python-pip". > > Sounds sane to me! > >> We can add that to the PEP as an alternative approach for redistributors to >> take that still offers a reasonable end-user experience. > > That'd really be great, and I'm sure it'd cut off the small bit of > concern some Debian-folk have about this stuff :) This also sounds reasonable to me. Really the PEP is about reducing user friction, especially for new users, so the ideal method is to ensure it's always installed but it'd be totally OK to do what Nick suggested as that still at leasts lets pypi packages to simply document installing as ``pip install `` and if it's not installed by default on Debian they'll get a good message telling them what they need to do. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 453 Round 4 - Explicit bootstrapping of pip in Python installations
On Sep 19, 2013, at 9:50 AM, Antoine Pitrou wrote: > Le Thu, 19 Sep 2013 09:27:24 -0400, > Donald Stufft a écrit : >> We've updated PEP453 based on some of the early feedback we've gotten >> from -dev and Martin. >> >> Major changes: >> >> * Removal of the option to fetch pip from PyPI in order not to modify >> the trust model of the Python installers >> * Consequently rename the model from ``getpip`` to ``extractpip`` > > "ensurepip" ? > > What happens if there is already a higher pip version installed? > I suppose "extractpip" doesn't do anything in that case? > > (sorry, perhaps it's mentioned in the PEP and I haven't seen it. > The PEP has become so long that I've only skimmed through it.) We could explicitly call this out but it's implied it will do nothing because it's going to just call out to to the privately installed pip to actually install itself, and pip itself won't "upgrade" to an older version. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 453 Round 4 - Explicit bootstrapping of pip in Python installations
On Sep 19, 2013, at 9:43 AM, Paul Moore wrote: > On 19 September 2013 14:27, Donald Stufft wrote: >> Major changes: >> >> * Removal of the option to fetch pip from PyPI in order not to modify the >> trust model of the Python installers >> * Consequently rename the model from ``getpip`` to ``extractpip`` > > If extractpip (I agree, I don't like the name, installpip is better) > only ever unpacks the bundled pip, and it's always run, why bother? > Why not just bundle pip directly into site-packages? The extra step > seems to add little or no value. > > Paul Well it's not always run (the PEP has it as an option in the installers that is checked by default) but even if it were always installed: - Nick has stated something about making it clear in the OSs installer DB which files are owned by Python and which are owned by pip - Upgrading becomes harder, instead of simply using pip's own mechanism the installer needs to take care not to clobber a user installed pip that is even newer than the bundled version. - "Fixing" a broken environment. If someone accidentally uninstalls pip this provides a simple way to reinstall it that doesn't require the old standby of getpip.py - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 453 Round 4 - Explicit bootstrapping of pip in Python installations
On Sep 19, 2013, at 9:36 AM, Paul Tagliamonte wrote: > On Thu, Sep 19, 2013 at 09:27:24AM -0400, Donald Stufft wrote: >> Rationale >> = >> >> Currently, on systems without a platform package manager and repository, >> installing a third-party Python package into a freshly installed Python >> requires first identifying an appropriate package manager and then >> installing it. > > Howdy Donald, > > Thanks for helping make Python better. > > However, speaking as a Debian-folk (but not for the Debian-folk, I'll > let barry take care of that), many users of Python software don't even > know what pip is (or even that Python exists) -- as such, I find it very > unlikely that a development tool would be shiped as part of the Python > distribution in Debian. I don't see this changing, even with this pep. > python-pip is still installable, but I don't see it pulled in by > default. That is obviously Debian's right. Hopefully if Debian does *not* pull it in by default they'll be the odd man out and documentation can be updated to say that on Debian based systems additional steps to get the "standard" Python install must be taken. I believe Fedora has +1'd this so if this gets accepted there will likely be that additional difference. > > >> Even on systems that *do* have a platform package manager, it is unlikely to >> include every package that is available on the Python Package Index, and > > Yes. This is true. However, it's not Debian's job to include all of > pypi. Pypi allows anyone to upload, and we have quite a bit of attention > to concerns such as providing source for everything (less of a concern > for Python, but pickles can still contain data not in the prefered form > of modification), and proper free software licensing. > > We also have a concern about stability; so we manage all the package set > together to allow non-technical end-users to install stuff and not worry > about breaking their system. It's not always the case where you can > upgrade a library without breaking API. This statement isn't meant to imply that it's Debian's job to include all of PyPI, it merely calls out the fact that people often do wish to install things that are not available in an OS repository while they are available on PyPI. > > > I'm trimming the rest, since I don't want to get dragged into > side-conversations about pip as a package manager. > > > Cheers, > Paul > > > -- > .''`. Paul Tagliamonte > : :' : Proud Debian Developer > `. `'` 4096R / 8F04 9AD8 2C92 066C 7352 D28A 7B58 5B30 807C 2A87 > `- http://people.debian.org/~paultag - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 453 Round 4 - Explicit bootstrapping of pip in Python installations
On Sep 19, 2013, at 9:27 AM, Donald Stufft wrote: > We've updated PEP453 based on some of the early feedback we've gotten from > -dev and Martin. > > Major changes: > > * Removal of the option to fetch pip from PyPI in order not to modify the > trust model of the Python installers > * Consequently rename the model from ``getpip`` to ``extractpip`` > > > Also available online at: http://www.python.org/dev/peps/pep-0453/ > > [snip] One thing I don't like is the name ``extractpip``, maybe ``installpip`` would be better? Naming things is hard. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 453 Round 4 - Explicit bootstrapping of pip in Python installations
ing the packaging tools to progress separately from the Python release and adoption schedules allows the improvements to be used by *all* members of the Python community and not just those able to live on the bleeding edge of Python releases. There have also been issues in the past with the "dual maintenance" problem if a project continues to be maintained externally while *also* having a fork maintained in the standard library. Since external maintenance of ``pip`` will always be needed to support earlier Python versions, the proposed bootstrapping mechanism will becoming the explicit responsibility of the CPython core developers (assisted by the pip developers), while pip issues reported to the CPython tracker will be migrated to the pip issue tracker. There will no doubt still be some user confusion over which tracker to use, but hopefully less than has been seen historically when including complete public copies of third-party projects in the standard library. Finally, the approach described in this PEP avoids some technical issues related to handle CPython maintenance updates when pip has been independently updated to a more recent version. The proposed pip-based bootstrapping mechanism handles that automatically, since pip and the system installer never get into a fight about who owns the pip installation (it is always managed through pip, either directly, or indirectly via the getpip bootstrap module). Defaulting to --user installation - Some consideration was given to bootstrapping pip into the per-user site-packages directory by default. However, this behavior would be surprising (as it differs from the default behavior of pip itself) and is also not currently considered reliable (there are some edge cases which are not handled correctly when pip is installed into the user site-packages directory rather than the system site-packages). .. _Wheel: http://www.python.org/dev/peps/pep-0427/ .. _pip: http://www.pip-installer.org .. _setuptools: https://pypi.python.org/pypi/setuptools .. _PEP439: http://www.python.org/dev/peps/pep-0439/ References == .. [#ubuntu] `Ubuntu <http://www.ubuntu.com/>` .. [#debian] `Debian <http://www.debian.org>` .. [#fedora] `Fedora <https://fedoraproject.org/>` .. [#homebrew] `Homebrew <http://brew.sh/>` .. [#conda] `Conda <http://www.continuum.io/blog/conda>` - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 453: Explicit bootstrapping of pip
On Sep 17, 2013, at 12:25 PM, "R. David Murray" wrote: > This PEP looks great to me. Thanks to everyone involved. > > On Wed, 18 Sep 2013 00:46:01 +1000, Nick Coghlan wrote: >> Reducing the burden of actually installing a third-party package should >> also decrease the pressure to add every useful module to the standard >> library. This will allow additions to the standard library to focus more >> on why Python should have a particular tool out of the box instead of >> using the general difficulty of installing third-party packages as >> justification for inclusion. > > I don't think this PEP changes the inclusion calculus, because I don't > think we've given any real weight to that in stdlib inclusion decisions. > I think the decision making is already focused on why Python should have > a particular tool out of the box. When I originally wrote this statement I wasn't trying to say that modules were included because of the lack of an easy installer, but instead that a not insignificant number of people had wanted to include stuff and some of their reasoning that I had seen in the past included the difficulty in bootstrapping the installer. So it decreases the *pressure* from others to include stuff for connivence not that it necessarily changes how python-dev itself weighs the decision to do so (but it might for some people?). > > The only context I recall where "installation difficulty" has been > considered relevant is the issue of corporate users restricted from > installing 3rd party packages...which will not be affected at all by > this PEP. > > --David > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/donald%40stufft.io - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Offtopic: OpenID Providers
On Sep 10, 2013, at 11:08 AM, Guido van Rossum wrote: > Why do several posts in this thread have an Unsubscribe link that tries to > unsubscribe me from the list? (I saw one by Glen, and another one by Donald > Stufft.) > > (Come to think of it, what's the point of having an Unbub link in ever > message that goes out?) All posts should have that. It's possible your mail client is collapsing it in some cases though because it looks like quoted text. > > > On Tue, Sep 10, 2013 at 12:21 AM, Glenn Linderman > wrote: > On 9/6/2013 10:22 AM, Dan Callahan wrote: >> On 9/5/13 12:31 PM, Jesus Cea wrote: >>> I have big hopes for Mozilla Persona, looking forward >>> Python infrastructure support :). >> >> Hi, I'm the project lead on Persona signin, and I spoke at PyCon earlier >> this year regarding why and how Mozilla is building Persona. If you'd like >> some more background, that video [0] is worth a look. >> >> Let's pull this discussion up a level: >> >> It sounds like many people (Jesus, Donald, Toshio, Barry, Tres, Dirkjan, >> etc.) are interested in seeing Persona on Python.org properties, and most of >> the objections coming from a place of "Persona hasn't gone viral, what if >> this is wasted effort?" > > OK, let's pull this discussion down a level: testing it out. > > So I tried to login to the crossword.thetimes.co.uk -- I used an email > address persona had never seen, it asked me for a password, and sent me a > confirmation message, containing a link that I clicked on. > > However, as I was reading clues and filling in blanks, I got a popup that > said "login failure [object Object]". And crossword told me it was saving > locally, and to login to save to the server. And the Log in button stayed > displayed, rather than a Log out button, which I assume it might get replaced > with if I ever get successfully logged in. > > Firefox 23.0.1, Windows 7 64-bit with autoupdates. Need any other info? Write > me privately if you want the email address I used (not the one I use here), > or the password. I used new ones, so I can share for testing, and then > discard them and use different ones "for real". If the system actually works. > Hey, the video demos looked great... > > Glenn > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > > > > > -- > --Guido van Rossum (python.org/~guido) - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Offtopic: OpenID Providers
On Sep 6, 2013, at 3:34 PM, "R. David Murray" wrote: > On Fri, 06 Sep 2013 15:17:12 -0400, Donald Stufft wrote: >> On Sep 6, 2013, at 3:11 PM, "R. David Murray" wrote: >> >>> IMO, single signon is overrated. Especially if one prefers not to make >>> it easy for various accounts to be automatically associated with one >>> another by various entities who shall remain nameless but have been in >>> the news a lot lately :) >> >> If I recall Persona doesn't leak this data like OpenID does, but >> perhaps Dan can speak to that better than I can. > > Note that I said that single signon *itself* was overrated. If you use > the same token to authenticate to multiple sites (and here the 'token' > is the email address) then your identities on those sites are ipso facto > associated with each other. *If* that email address is also never > leaked (never displayed, even to other signed on users, all communication > with the site encrypted), then you only have to worry if the > sites exchange information about their accounts, or if the government > comes knocking on their doors > > Yes, I'm paranoid. That doesn't mean they aren't listening. > > That said, sometimes you *want* identities to be associated, so I'm not > saying Persona is a bad thing. Just that single signon is overrated. Well that's fine to have that opinion but I think you're under estimating just how easy it is to link two disparate accounts especially if you have the cooperation (willing or otherwise) of the site operators. I've personally seen Google do some particularly amazing connections between accounts that I don't believe using the same authentication token is going to make that any easier or harder for them. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Offtopic: OpenID Providers
On Sep 6, 2013, at 1:22 PM, Dan Callahan wrote: > On 9/5/13 12:31 PM, Jesus Cea wrote: >> I have big hopes for Mozilla Persona, looking forward >> Python infrastructure support :). > > Hi, I'm the project lead on Persona signin, and I spoke at PyCon earlier this > year regarding why and how Mozilla is building Persona. If you'd like some > more background, that video [0] is worth a look. > > Let's pull this discussion up a level: > > It sounds like many people (Jesus, Donald, Toshio, Barry, Tres, Dirkjan, > etc.) are interested in seeing Persona on Python.org properties, and most of > the objections coming from a place of "Persona hasn't gone viral, what if > this is wasted effort?" > > We can tackle that from two angles: > > 1. Dirkjan and I are willing to do the work to make this happen if someone > from python-devel is willing to guide us through the contributor process for > these systems. FWIW I'm a maintainer of PyPI and I do plan on enabling Persona there. Mostly blocked because I want to focus my PyPI efforts on the "next gen" code base instead. > > 2. There's a seamless migration path away from Persona if we fail: fall back > to the pre-existing traditional email/password system using the same email > addresses that Persona had previously been in charge of verifying. > > So let's do this. The open web deserves better than just Google+, Facebook, > or Passwords, and visible support from the Python community would be a huge > step toward answering the chicken-and-egg objections raised in this thread. > > At your service, > -Callahad > > PS: Freeform OpenID has utterly failed as a user-empowering authentication > system, and the protocol itself is rapidly being supplanted by > vendor-specific OAuth[1] systems. If we want to ensure that "you *can* (not > *must*) use free and open services to access our resources," then we must > provide an option to use something akin to Persona. > > [0]: http://pyvideo.org/video/1764 > > [1]: "Google's OpenID service is being replaced by Login with OAuth 2.0." > https://developers.google.com/accounts/docs/GettingStarted > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/donald%40stufft.io - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Offtopic: OpenID Providers
On Sep 6, 2013, at 3:11 PM, "R. David Murray" wrote: > IMO, single signon is overrated. Especially if one prefers not to make > it easy for various accounts to be automatically associated with one > another by various entities who shall remain nameless but have been in > the news a lot lately :) If I recall Persona doesn't leak this data like OpenID does, but perhaps Dan can speak to that better than I can. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Offtopic: OpenID Providers
On Sep 5, 2013, at 5:56 PM, Oleg Broytman wrote: > On Thu, Sep 05, 2013 at 05:29:07PM -0400, Barry Warsaw > wrote: >> I don't want my choice to be log into Facebook or manage a slew of passwords. > > The last part is unavoidable. I regularly login to LiveJournal, > Twitter, SourceForge, BitBucket, Gitorious, GitHub and to hundreds of > other sites -- blogs, torrents, web shops. I already manage hundreds of > passwords. OpenID promised to save me from that and failed. Do you think > Persona would succeed in this regard (saved me from managing all those > passwords)? And if not -- what are the benefits? I already manage > hundreds of passwords -- two or three additional passwords for > bugs.python.org, wiki.python.org and so on don't make the situation > worse. > IMO the very idea of single sign-on in the open web is meaningless. > >> But I'm not volunteering to do the work, so I don't get to decide. I'm just >> stating that I think our principle should be that you *can* (not *must*) use >> free and open services to access our resources. > > Well, I can only use services that are available, not those that are > promised. If python.org grows support for Persona -- who will be my > provider and for what price? I am not going to install and manage > additional software on my servers -- I don't want to be my own provider, > I have enough job already. Theoretically whoever runs the domain for your email address (since Persona uses email as your identifier). In order to make it work as a stop gap they also have more openid like idP's which they run a major one (that also offers a "bridge" to make things like Gmail work on Persona without needing to register for anything else). > > Oleg. > -- > Oleg Broytmanhttp://phdru.name/p...@phdru.name > Programmers don't die, they just GOSUB without RETURN. > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/donald%40stufft.io - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Offtopic: OpenID Providers
On Sep 5, 2013, at 4:53 PM, Barry Warsaw wrote: > On Sep 06, 2013, at 12:36 AM, Oleg Broytman wrote: > >> You cannot login using OpenID to most interesting popular sites. >> GMail? No. Twitter? No. Facebook? FriendFeed? identi.ca? No, no, no. > > I'd be surprised if you ever saw the big social networking sites support > OpenID or Persona. They want to own that space themselves, so probably have > no business incentive to support 3rd party systems. Not that it changes this statement at all but you wouldn't expect to see a Persona login for gmail as persona solves the problem that people don't think of urls as personal identifiers by replacing it with emails. So Gmail would be the Persona IdP > > We're open source, and I think it benefits our mission to support open, > decentralized, and free systems like OpenID and Persona. > > -Barry > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/donald%40stufft.io - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Offtopic: OpenID Providers
On Sep 5, 2013, at 2:43 PM, Oleg Broytman wrote: > On Thu, Sep 05, 2013 at 02:35:16PM -0400, Donald Stufft > wrote: >> Persona is the logical successor to OpenID. > > OpenID lived a short life and died a quiet death. I'm afraid Persona > wouldn't live even that much. Dead-born idea, in my so humble opinion. I don't think there's much evidence to support this. I'm seeing more sites support Persona not less. It solves some of the major problems with OpenID. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Offtopic: OpenID Providers
On Sep 5, 2013, at 2:25 PM, Oleg Broytman wrote: > On Thu, Sep 05, 2013 at 02:16:29PM -0400, Donald Stufft > wrote: >> >> On Sep 5, 2013, at 2:12 PM, Oleg Broytman wrote: >>> I used to use myOpenID and became my own provider using poit[1]. >>> These days I seldom use OpenID -- there are too few sites that allow >>> full-featured login with OpenID. The future lies in OAuth 2.0. >> >> The Auth in OAuth stands for Authorization not Authentication. > > There is no authorization without authentication, so OAuth certainly > performs authentication: http://oauth.net/core/1.0a/#anchor9 , > http://tools.ietf.org/html/rfc5849#section-3 They are separate topics and authorization does not need to imply authentication, it so happens that in many particular instances of OAuth you can estimate authentication. https://en.wikipedia.org/wiki/OAuth#OpenID_vs._pseudo-authentication_using_OAuth Persona is the logical successor to OpenID. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Offtopic: OpenID Providers
On Sep 5, 2013, at 2:12 PM, Oleg Broytman wrote: > On Thu, Sep 05, 2013 at 07:31:59PM +0200, Jesus Cea wrote: >> I just received an email from my OpenID provider, "myOpenID", saying >> that they drop OpenID service next February. I wonder what other >> OpenID providers are used by other python-dev fellows. >> >> What are you using?. bugs.python.org admins could share some data? >> >> I agree than OpenID is (quite) dead, but I rather prefer OpenID to use >> user/pass. I have big hopes for Mozilla Persona, looking forward >> Python infrastructure support :). >> >> PS: I use "http://www.jcea.es/"; as my OpenID identity, and I delegate >> the actual service to "myOpenID". I can switch delegation trivially. > > I used to use myOpenID and became my own provider using poit[1]. > These days I seldom use OpenID -- there are too few sites that allow > full-featured login with OpenID. The future lies in OAuth 2.0. The Auth in OAuth stands for Authorization not Authentication. > > 1. http://yangman.ca/poit/ > > Oleg. > -- > Oleg Broytmanhttp://phdru.name/p...@phdru.name > Programmers don't die, they just GOSUB without RETURN. > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/donald%40stufft.io - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] When to remove deprecated stuff
On Aug 22, 2013, at 1:34 PM, Ezio Melotti wrote: > Hi, > > On Thu, Aug 22, 2013 at 1:00 PM, Petri Lehtinen wrote: >> >> Removing some cruft on each release can be very painful for users. >> >> Django's deprecation policy works like this: They deprecate something >> in version A.B. It still works normally in A.B+1, generates a >> (silenced) DeprecationWarning in A.B+2, and is finally removed in >> A.B+3. > > I see two problems with this: > 1) DeprecationWarnings should be generated as soon as the feature is > deprecated (i.e. A.B, not A.B+2). Holding off the warnings is not > helping anyone. > 2) The deprecation period seems fixed and independent from the > feature. IMHO the period should vary depending on what is being > deprecated. Little known/used "features" could be deprecated in A.B > and removed in A.B+1; common "features" can be deprecated in A.B and > removed in A.B+n, with an n >= 2 (or even wait for A+1). This isn't exactly accurate, it raises a PendingDeprecation warning in A.B, Deprecation in A.B+1, and removed in A.B+2. PendingDeprecation (In Django) was designed to be silent by default and Deprecation loud by default. That got messed up when Python silenced Deprecation warnings by default and we've had to resort to some ugliness to restore that behavior. > >> So if I haven't read through the full release notes of each >> release nor enabled DeprecationWarnings, I end up having something >> broken each time I upgrade Django. >> > > Reading the release notes shouldn't be required -- DeprecationWarnings > are already supposed to tell you what to change. > If you have good test coverage this should happen automatically (at > least with unittest), but even if you don't you should run your code > with -Wa before upgrading (or test your code on the new version before > upgrading Python/Django/etc. in production). > > Best Regards, > Ezio Melotti > >> I hope the same will not start happening each time I upgrade Python. >> When the removals happen on major version boundaries, it should be >> more evident that something will break. Then people can enable >> DepreationWarnings and test all the affected code thoroughly with the >> new version before upgrading. >> >> Petri > ___________ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/donald%40stufft.io - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] backported Enum
On Jun 16, 2013, at 1:52 AM, Ben Finney wrote: > Donald Stufft writes: > >> On Jun 15, 2013, at 10:45 PM, Ben Finney wrote: >>> Is there anything I can do to keep the ‘enum’ package online for >>> continuity but make it clear, to automated tools, that this is >>> end-of-life and obsoleted by another package? >> >> Right now the best you can do is make a note of it. Pep 426 let's you >> do what you want. > > Thanks. What specifically in PEP 426 do you mean; how would I make a > note of “this package is end-of-life as is, please migrate to > ‘flufl.enum’ instead” using PEP 426 metadata? http://www.python.org/dev/peps/pep-0426/#obsoleted-by Note PEP426 isn't completed and isn't implemented :) > > -- > \“There are no significant bugs in our released software that | > `\ any significant number of users want fixed.” —Bill Gates, | > _o__) 1995-10-23 | > Ben Finney > > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/donald%40stufft.io - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] backported Enum
On Jun 15, 2013, at 10:45 PM, Ben Finney wrote: > Ethan Furman writes: > >> So I have the stdlb 3.4 Enum backported for both earlier 3.x and back >> to 2.4 in the 2.x series. >> >> I would like to put this on PyPI, but the `enum` name is already >> taken. > > I have for a long time approved of ‘flufl.enum’ becoming the One Obvious > Way to do enumerations, and am glad to see PEP 435 bring it into the > standard library. > > http://mail.python.org/pipermail/python-ideas/2011-July/010811.html> > > There are some people still contacting me about their use of the ‘enum’ > package on PyPI, so I think that to just replace it with a different > code base under the same name would not be helpful. > >> Would it be inappropriate to call it `stdlib.enum`? > > That sounds good to me. > > Is there anything I can do to keep the ‘enum’ package online for > continuity but make it clear, to automated tools, that this is > end-of-life and obsoleted by another package? Right now the best you can do is make a note of it. Pep 426 let's you do what you want. > > -- > \ “Free thought is a necessary, but not a sufficient, condition | > `\ for democracy.” —Carl Sagan | > _o__) | > Ben Finney > > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/donald%40stufft.io ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] backported Enum
On Jun 15, 2013, at 5:43 PM, Ethan Furman wrote: > On 06/15/2013 01:53 PM, Antoine Pitrou wrote: >> On Sat, 15 Jun 2013 12:46:32 -0700 >> Ethan Furman wrote: >>> So I have the stdlb 3.4 Enum backported for both earlier 3.x and back to >>> 2.4 in the 2.x series. >>> >>> I would like to put this on PyPI, but the `enum` name is already taken. >>> >>> Would it be inappropriate to call it `stdlib.enum`? >> >> `backport.enum`? > > Some other well-meaning soul has already claimed that name, but that one only > covers 2.6 and 2.7. > > -- > ~Ethan~ > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/donald%40stufft.io I claimed backport.enum, but you're welcome to the name. I was going to try and backport this PEP under that name anyways. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Clean way in python to test for None, empty, scalar, and list/ndarray? A prayer to the gods of Python
I never want to iterate, but I love slice syntax and indexing. Don't think you can have that w/o being able to loop over it can you? Maybe I'm just thinking slow since I just woke up. On Jun 15, 2013, at 8:53 AM, Tres Seaver wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > On 06/14/2013 04:55 PM, R. David Murray wrote: >> This I sort of agree with. I've often enough wanted to know if >> something is a non-string iterable. But you'd have to decide if >> bytes/bytearray is a sequence of integers or a scaler... > > In fifteen years of Python programming, I have literally *never* wanted > to iterate over 'str' (or now 'bytes'). I've always considered the fact > that Python made them iterable by default (rather than e.g. defining a > method / property to get to an iterable "view" on the underlying string) > a wart and a serious bug magnet. > > > Tres. > - -- > === > Tres Seaver +1 540-429-0999 tsea...@palladion.com > Palladion Software "Excellence by Design"http://palladion.com > -BEGIN PGP SIGNATURE- > Version: GnuPG v1.4.11 (GNU/Linux) > Comment: Using GnuPG with undefined - http://www.enigmail.net/ > > iEYEARECAAYFAlG8Y7MACgkQ+gerLs4ltQ6EbwCfYlC3JKL22HK7WgxJLAh9Gk2H > R4cAn2+ijAkebHuF92txeddBq99L/zqn > =bLkO > -END PGP SIGNATURE- > > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/donald%40stufft.io ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PyPI upload error
On Jun 4, 2013, at 9:47 PM, MRAB wrote: > I've just tried to upload to PyPI using setup.py and got this error: > >Upload failed (503): backend write error > > Can anyone tell me what it means? > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/donald%40stufft.io This is probably more appropriate for distutils-sig, but does it happen every time? or did it just happen once? - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython)
On Jun 3, 2013, at 6:01 PM, Paul Moore wrote: > > On 3 June 2013 22:46, Donald Stufft wrote: >> Also, we should consider the issue for application users. Suppose I'm using >> a Python application that downloads something from the web. I upgrade to >> 3.4, and the app stops working because of a "will cease to work" case. As an >> end user, how can I get the app working again? Having to patch the sources >> isn't an option, and reverting to 3.3 provokes the reaction "Python broke my >> app". > > Supply a SSL vert using the environment variable? > > Hmm, that would be acceptable, I guess, for many users (although Windows > users are somewhat more environment-variable-averse than Unix users). But you > say that as if it's obvious how to do that (or where to get a cert). It's > certainly not obvious to me, and if "it works in Internet Explorer", I'd have > no idea where to get a cert from that I could use in an environment variable. > > Just to repeat - I agree with the principle, but in many environments, users > are pretty much clueless about security and actively object to being educated > "for their own safety". These users will disable all security quite happily > if it stops the internal app failing, and will blame Python for "making > things harder" and breaking backward compatibility. On the other hand, I > suspect we're talking about an extremely low percentage of cases, so let's > not blow the issue out of proportion :-) > > Paul Let me make myself a bit more clear because maybe I haven't been. If someone is relying on the insecure by default behavior and Python 3.4 gets this change, their code will break and their options will be: Client side: - Disable Verification, essentially reverting back to the old behavior. - Setup their Python install to be able to validate that certificate - This may be importing it into their OS certificate store - This may be using an env var to specify - This may be passing it directly somewhere Server side: - Make the certificate valid - Purchase a Valid by default certificate - Serve immedaries etc. Generally any of these will be completely valid options, even disabling the checks. The idea behind my proposal is that people generally only use TLS for a reason and that reason is they want to protect against the kinds of attacks that TLS protects against. You really only get the bulk of those protections if you validate the certificates. So for the vast bulk of people validation is the option they want. It also happens that validation on by default is "secure" by default which aims a fairly large foot gun away from peoples feet. To expand on something else, I'm completely willing to do whatever I can to hep make this happen. I don't really know what the process is. If I need to write a PEP I'll do that. If I need to volunteer to manage the certificates, I'll do that. If we need docs or tooling to help the people who this change might break I'll do that too. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython)
On Jun 3, 2013, at 5:51 PM, Antoine Pitrou wrote: > On Mon, 3 Jun 2013 17:47:31 -0400 > Donald Stufft wrote: >> >> On Jun 3, 2013, at 5:41 PM, Antoine Pitrou wrote: >> >>> On Mon, 3 Jun 2013 22:31:40 +0100 >>> Paul Moore wrote: >>>> >>>>> Some legit sites with proper >>>>> certificates still manage to muck something up administratively >>>>> (developer.quicksales.com.au has a cert from RapidSSL but doesn't >>>>> bundle the intermediates, and I've told their devs about it, but all I >>>>> can do is disable cert checking). This will break code in ways that >>>>> will surprise people greatly. But I'd still rather the default be >>>>> True. >>>>> >>>> >>>> I'm happy if the "will cease to work" clause only says "some sites with >>>> broken security configurations may stop working" with a clear explanation >>>> that it is *their* fault, not Python's. I'd also expect that the same sites >>>> would fail in browsers - if not, we should also be able to make them work >>>> (or face cries of "well, Internet Explorer/Firefox doesn't have a problem >>>> with my site, why does Python?"). >>> >>> Keep in mind that not every HTTPS service is a Web site that is meant >>> to be readable with a browser. Some are Web services, possibly internal, >>> possibly without a domain name (and, therefore, probably a non-matching >>> certificate subject name). >> >> They should need to explicitly opt in to disabling the checks that allow >> that to work. > > Obviously, which means compatibility is broken with existing code. > > Regards > > Antoine. > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/donald%40stufft.io Yes in that case compat will be broken and they'll need to either specify a cert that can be used to validate the connection or disable the protection. I think it's very surprising for people that they need to *enable* secure mode when most tools have that on by default. It's handing users a security foot gun, and like most things security related "broken" is silent until it's too late. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython)
On Jun 3, 2013, at 5:31 PM, Paul Moore wrote:
>
> On 3 June 2013 21:05, Chris Angelico wrote:
> +1 for having the default be safe, but this will have to be very
> loudly announced ("when migrating from 3.3 to 3.4, TLS connections
> will cease to work if blah blah").
>
> +1 on the default being safe, certainly. But with the proviso that the same
> code should work in 3.3 and 3.4, with no user impact (other than that the
> connection is safer, but that's not user-visible unless there's an attack :-))
If we bundle certs that will be the case sans connections where it doesn't
validate.
>
> In other words, that "will cease to work" clause should not exist - but see
> below...
>
> Some legit sites with proper
> certificates still manage to muck something up administratively
> (developer.quicksales.com.au has a cert from RapidSSL but doesn't
> bundle the intermediates, and I've told their devs about it, but all I
> can do is disable cert checking). This will break code in ways that
> will surprise people greatly. But I'd still rather the default be
> True.
>
> I'm happy if the "will cease to work" clause only says "some sites with
> broken security configurations may stop working" with a clear explanation
> that it is *their* fault, not Python's. I'd also expect that the same sites
> would fail in browsers - if not, we should also be able to make them work (or
> face cries of "well, Internet Explorer/Firefox doesn't have a problem with my
> site, why does Python?").
Browsers tend to download intermediate certs while I don't think Python does.
>
> Also, we should consider the issue for application users. Suppose I'm using a
> Python application that downloads something from the web. I upgrade to 3.4,
> and the app stops working because of a "will cease to work" case. As an end
> user, how can I get the app working again? Having to patch the sources isn't
> an option, and reverting to 3.3 provokes the reaction "Python broke my app".
Supply a SSL vert using the environment variable?
>
> Summary - I'm +1 as long as either the "will cease to work" list is empty, or
> we have a *very* good story for (legitimate) sites which do cease to work.
>
> Paul.
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/donald%40stufft.io
-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython)
On Jun 3, 2013, at 5:41 PM, Antoine Pitrou wrote: > On Mon, 3 Jun 2013 22:31:40 +0100 > Paul Moore wrote: >> >>> Some legit sites with proper >>> certificates still manage to muck something up administratively >>> (developer.quicksales.com.au has a cert from RapidSSL but doesn't >>> bundle the intermediates, and I've told their devs about it, but all I >>> can do is disable cert checking). This will break code in ways that >>> will surprise people greatly. But I'd still rather the default be >>> True. >>> >> >> I'm happy if the "will cease to work" clause only says "some sites with >> broken security configurations may stop working" with a clear explanation >> that it is *their* fault, not Python's. I'd also expect that the same sites >> would fail in browsers - if not, we should also be able to make them work >> (or face cries of "well, Internet Explorer/Firefox doesn't have a problem >> with my site, why does Python?"). > > Keep in mind that not every HTTPS service is a Web site that is meant > to be readable with a browser. Some are Web services, possibly internal, > possibly without a domain name (and, therefore, probably a non-matching > certificate subject name). They should need to explicitly opt in to disabling the checks that allow that to work. > > Regards > > Antoine. > > > ___________ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/donald%40stufft.io - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython)
On Jun 3, 2013, at 4:19 PM, Christian Heimes wrote: > Am 03.06.2013 21:52, schrieb Antoine Pitrou: >> cadefault=True will probably be fail if the system certs are not >> properly configured in OpenSSL, e.g. under Windows or with a hand-made >> OpenSSL build. >> And, because of the way the OpenSSL API works, there's no way of >> knowing if it is the case or not: >> http://docs.python.org/3.4/library/ssl.html#ssl.SSLContext.set_default_verify_paths > > I only see an issue for uncommon Linux distributions and exotic Unices. > > For Windows an interface to crypt32 API solves the CA issue as shown in > my wincertstore module. It gives the user the same SSL experience as > Internet Explorer. > > Most Linux and BSD-ish operating systems have SSL certs at some standard > location. > https://bitbucket.org/pypa/setuptools/src/6de3186fdfd9f5b543380e9aca2d48976cfc38cd/setuptools/ssl_support.py?at=default#cl-15 > lists a couple of standard locations. > > Under which conditions do we need to ship a CA cert file? > > Christian > > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/donald%40stufft.io What about OSX? - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython)
On Jun 3, 2013, at 1:07 PM, Barry Warsaw wrote: > On Jun 03, 2013, at 02:17 PM, Donald Stufft wrote: > >> I'd actually prefer for Linux to not use the bundled certs when installed >> from a package manager because it should use the system certs, but people >> can't depend on certs being there if they are only there on linux. > > I think we agree on that. > >> Adding them into Python means people _can_ depend on them being there, and >> Windows and other systems without system integrators to modify it to use the >> system store will still get certs and Ubuntu can make it just work(™). > > Again, I think PEP 431 provides a pretty good model for how this should be > done. Maybe it's worth factoring out this specific part of PEP 431 into an > informational PEP? Looks fine to me minus the not updating in security releases (but that's just a difference in the type of data). > >> This would probably (eventually) make the bundling of certificates better >> too. >> >> Meaning that once it's been in long enough people are willing to depend on >> it, they won't need to bundle their own certs and ubuntu/debian can just >> modify the one location instead of needing to modify it for every package >> that does it. > > Can we do the same for the JavaScript libraries? :) > > -Barry - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython)
On Jun 3, 2013, at 12:48 PM, Barry Warsaw wrote: > On Jun 03, 2013, at 09:05 AM, Ben Darnell wrote: > >> The data is analogous to the time zone database (PEP 431) in that it may >> need to be updated independently of Python's own release schedule, so we >> may want to use similar techniques to manage both. Also see certifi ( >> https://pypi.python.org/pypi/certifi), which is a copy of the Mozilla list >> in a pip-installable form. > > Right, this is very much analogous, except with the additional twist that > out-of-date certificates can pose a significant security risk. > > I'm fairly certain that Debian and Ubuntu would explicitly not use any > certificates shipped with Python, for two main reasons: 1) our security teams > already manage the certificate store distro-wide and we want to make sure that > one update fixes everything; 2) we don't want to duplicate code in multiple > packages[1]. > > So *if* Python decides to do this (and I'm -0, but from a decidedly > Linux-distro bias), it must be easily disabled. I generally like the way PEP > 431 handles the tzdata, so I think we should do the same here. I'd actually prefer for Linux to not use the bundled certs when installed from a package manager because it should use the system certs, but people can't depend on certs being there if they are only there on linux. Adding them into Python means people _can_ depend on them being there, and Windows and other systems without system integrators to modify it to use the system store will still get certs and Ubuntu can make it just work(™). > > -Barry > > [1] This gives us headaches in upstreams like coverage caused by bundling > externally available JavaScript libraries, or like urllib3 bundling chardet > and urllib3, not to mention their own certificates yet again. :( This would probably (eventually) make the bundling of certificates better too. Meaning that once it's been in long enough people are willing to depend on it, they won't need to bundle their own certs and ubuntu/debian can just modify the one location instead of needing to modify it for every package that does it. > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/donald%40stufft.io - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython)
On Jun 3, 2013, at 12:52 PM, Barry Warsaw wrote: > On Jun 03, 2013, at 03:12 AM, Donald Stufft wrote: > >> That's fine with me too. My only reason for wanting to use the system certs >> first is so if someone has modified their system certs (say to include a >> corporate cert) that it would ideally take affect for Python as well. > > This reminds me of one other thing. We have to make sure that the APIs > (e.g urlopen()) continue to allow us to use self-signed certificates, if for > no other reason than for testing purposes. OTOH, taking this away would be a > backward incompatible change in API so probably wouldn't happen anyway. > > -Barry > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/donald%40stufft.io The other additional comment I'd like to throw in here is that if we don't bundle SSL certs I think we should still verify by default (which means HTTPS urls will throw an error by default if we can't locate a certificate store) because I think the risk to people unknowingly thinking that their HTTPS urls are protected are significant enough that this "error" shouldn't be silent by default. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython)
On Jun 3, 2013, at 12:36 PM, Barry Warsaw wrote: > On Jun 03, 2013, at 01:20 AM, Donald Stufft wrote: > >> So I would like to propose that CPython adopt the Mozilla SSL certificate >> list and include it in core, and switch over the API's so that they verify >> HTTPS by default. This is what most people are going to expect when using a >> https url (Especially after learning that Python 2.x doesn't verify TLS, but >> Python 3.x "does"). > > For the "verify HTTPS by default", do you mean specifically changing the > cadefault argument to urllib.request.urlopen() to True? Note that I recently > closed a bug related to this: > > http://bugs.python.org/issue17977 > > +1 for changing the default to True. > > -Barry > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/donald%40stufft.io If that's all it takes to make sure that people have to opt out of an invalid certificate throwing an error than yes :) My goal here is to make it that when someone accesses a TLS secured network resource (I said HTTP, but if there are other things, xmlrpclib, or what not where people can reasonably expect valid TLS certificates those too) they are protected by a MITM attack by default. I worry with the current situation people will just use TLS connections without realizing it's not being verified and thinking they are "safe". As with most things security the outcome of "failing" to do it right when the default is insecure is well nothing until someone attacks you. So Hopefully we make things secure by default for folks :) - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython)
On Jun 3, 2013, at 12:52 PM, Ethan Furman wrote: > On 06/03/2013 09:43 AM, Donald Stufft wrote: >> On Jun 3, 2013, at 5:51 AM, Antoine Pitrou wrote: >>> >>> The problem with a "slightly outdated" CA store is that it can be a >>> security risk. >> >> Tracking the Mozilla store isn't difficult. New additions can be ignored for >> currently released Pythons so we'd just >> need to watch them for blacklisting certs and roll that into a security >> update. > > Personally, I'm not interested in waiting six months for an update. And why > can't I have the new additions? > > Seems to me a better solution is to have routines that can query and update > at will (meaning the app has to call them), as well as having the bundle > (black lists as well as new additions) in the regular updates. > > -- > ~Ethan~ > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/donald%40stufft.io Personally I view that as a nice to have but if an app is willing to call hem they can easily depend on certifi and just replace the default certificates. The goal here is to get secure by default without any work on the part of the developers. If a developer knows to update the certs they can know to use certifi. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython)
On Jun 3, 2013, at 5:51 AM, Antoine Pitrou wrote: > On Mon, 3 Jun 2013 21:37:10 +1200 > Ben Hoyt wrote: >> >> I'm not familiar with Unix/Linux, but on Windows, if it's anything >> like mimetypes it'll be really hard to get consistent behaviour across >> different boxes/versions from the registry, or wherever certs might be >> stored on Windows. I'd much rather have a slightly outdated but >> consistent experience by default. > > The problem with a "slightly outdated" CA store is that it can be a > security risk. > > Regards > > Antoine. > > > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/donald%40stufft.io Tracking the Mozilla store isn't difficult. New additions can be ignored for currently released Pythons so we'd just need to watch them for blacklisting certs and roll that into a security update. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython)
On Jun 3, 2013, at 1:58 AM, Benjamin Peterson wrote: > 2013/6/2 Donald Stufft : >> As of right now, as far as I can tell, Python does not validate HTTPS >> certificates by default. As far as I can tell this is because there is no >> guaranteed certificates available. >> >> So I would like to propose that CPython adopt the Mozilla SSL certificate >> list and include it in core, and switch over the API's so that they verify >> HTTPS by default. > > +1 > >> >> Ideally this would take the shape of attempting to locate the system >> certificate store if possible, and if that doesn't work falling back to the >> bundled certificates. That way the various Linux distros can easily have >> their copies of Python depend soley on their built in certs, but Windows, >> OSX, Source compiles etc will all still have a fallback value. > > My preference would be actually be for the included certificates file > to be used by default. This would provide a consistent experience > across platforms. We could provide options to look for system cert > repositories if desired. That's fine with me too. My only reason for wanting to use the system certs first is so if someone has modified their system certs (say to include a corporate cert) that it would ideally take affect for Python as well. But honestly the Linux distros will probably modify things to use system certs anyways and non Linux (esp Windows) probably doesn't have a way to get those system certs into OpenSSL. > > > > -- > Regards, > Benjamin - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython)
As of right now, as far as I can tell, Python does not validate HTTPS certificates by default. As far as I can tell this is because there is no guaranteed certificates available. So I would like to propose that CPython adopt the Mozilla SSL certificate list and include it in core, and switch over the API's so that they verify HTTPS by default. This is what most people are going to expect when using a https url (Especially after learning that Python 2.x doesn't verify TLS, but Python 3.x "does"). Ideally this would take the shape of attempting to locate the system certificate store if possible, and if that doesn't work falling back to the bundled certificates. That way the various Linux distros can easily have their copies of Python depend soley on their built in certs, but Windows, OSX, Source compiles etc will all still have a fallback value. ----- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Why can't I encode/decode base64 without importing a module?
On Apr 22, 2013, at 10:04 PM, Steven D'Aprano wrote: > On 23/04/13 09:16, Greg Ewing wrote: >> Victor Stinner wrote: >>> The last proposition is to add transform() and untransform() methods >>> to bytes and str types. ... If I remember >>> correctly, the missing point is how to define which types are >>> supported by a codec >> >> Also, for any given codec, which direction is "transform" >> and which is "untransform"? >> >> Also also, what's so special about base64 et al that they >> deserve an ultra-special way of invoking them, instead of >> having to import a class or function like you do for >> *every* *other* piece of library functionality? > > > As others have pointed out in the past, repeatedly, the codec system is > completely general and can transform bytes->bytes and text->text just as > easily as bytes<->text. Or indeed any bijection, as the docs for 2.7 point > out. The question isn't "What's so special about base64?" The questions > should be: > > - What's so special about exotic legacy transformations like ISO-8859-10 and > MacRoman that they deserve a string method for invoking them? > > - Why have common transformations like base64, which worked in 2.x, been > relegated to second-class status in 3.x? > > - If it is no burden to have to import a module and call an external function > for some transformations, why have encode and decode methods at all? > > > If you haven't read this, you should: > > http://lucumr.pocoo.org/2012/8/11/codec-confusion/ I may be dull, but it wasn't until I started using Python 3 that it really clicked in my head what encode/decode did exactly. In Python2 I just sort of sprinkled one or the other when there was errors until the pain stopped. I mostly attribute this to str.decode and bytes.encode not existing. > > > > > -- > Steven > _______ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/donald%40stufft.io - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython (2.7): Issue 17538: Document XML vulnerabilties
On Mar 26, 2013, at 2:41 PM, Antoine Pitrou wrote: > On Tue, 26 Mar 2013 17:53:39 +0100 (CET) > christian.heimes wrote: >> + >> +The XML processing modules are not secure against maliciously constructed >> data. >> +An attacker can abuse vulnerabilities for e.g. denial of service attacks, to >> +access local files, to generate network connections to other machines, or >> +to or circumvent firewalls. > > Really? Where does the "access local files, generate network > connections, circumvent firewalls" allegation come from? https://pypi.python.org/pypi/defusedxml#external-entity-expansion-remote https://pypi.python.org/pypi/defusedxml#external-entity-expansion-local-file https://pypi.python.org/pypi/defusedxml#dtd-retrieval > > Regards > > Antoine. > > > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/donald%40stufft.io - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] built-in Python test runner
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Barry Warsaw wrote: > On Mar 05, 2013, at 02:11 AM, Donald Stufft wrote: > >> Doesn't setuptools/distribute already have a setup.py test command? >> That seems like the easiest way forward? > > Yes, and in theory it can make `python setup.py test` work well. But > there are lots of little details (such as API differences for > ensuring that doctests run, "additional tests" discovery, etc.) that > make this often not work so well in practice. Some of that is social > and some of it is technical. I still claim that including test suite > information in a package's metadata would be a win, but maybe that's > just too much to hope for right now. For the "right now" solution you can easily override the test command in setup.py. Long term this would be something that could be added to the METADATA but I think it is (and rightly should be) lower priority to actually getting it working for it's core purpose. > > -Barry ___ Python-Dev > mailing list Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: > http://mail.python.org/mailman/options/python-dev/donald%40stufft.io -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.18 (Darwin) iQIcBAEBAgAGBQJRNlaJAAoJEG48vOkzctz6SvsP/2+MYkn1NgcHdmaQg09A3MDH a++7+hGGsXgQwwJ3q3u6T6Hzd1pokJI7hqGHAXBnkODrZZ8f9Z0+OP9I8HPUmo0D mJDsDxn2VZImBkNZJlBGNFKIz4EjS5llFapzdl58ZkIVZ7Rz3PTaPVSiXOc+ppp7 xYkWXGnx+2DTFaHywi9qGsHrbAXqgcwIhXO29NYl98xA9X98/XSRiXmHVfNURA3U 1GV1G9H1Qfvu8YjfYfBCCUn6db6eLGVPO7VcxRh6Cyzfk5SFSuziCVI8v3t3msjw KSba+8Pe3RQ7RS17VEJqCMQjkhhGnAndgIL3Jho41qb3g+Rdk2OP+weWbYV92Q8F HL6QtPgm5/QS5tKyl6nK97+9q+NdhGOEzKOL0pBiF4HKdT0mKyBxqttVIgUDAVMQ XcjhBu1wnpQnhkeZ8F3yGNubmE2tRdVVfhTfVDaA3ICl7uVBlbtUUMTJRK7DQ4vW gzDk5aKJB8OHimC1ijeTQm3M2lXkS5z0e5IuaION2WrG5A2BEvH12d/I7ekc1Ixa lVhScABFwY0UcrMt4td65er/w4Z0S+BL87SZMH1mffoS6XmL3fxUuOtAB+MUWDk7 Rd4xXnUFvr3SMmMRjSogpO/HO5IBpuzUwu0wqSz8qPcex+l+lHyOEHBDvacEcv9b zvGYjfQuk+2hWlAqw5o4 =6ELp -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] built-in Python test runner (was: Python Language Summit at PyCon: Agenda)
On Tuesday, March 5, 2013 at 2:02 AM, Lennart Regebro wrote: > On Tue, Mar 5, 2013 at 1:41 AM, Robert Collins > mailto:robe...@robertcollins.net)> wrote: > > So that is interesting, but its not sufficient to meet the automation > > need Barry is calling out, unless all test suites can be run by > > 'python -m unittest discover' with no additional parameters [and a > > pretty large subset cannot]. > > > > > But can they be changed so they are? That's gotta be the important bit. > > What's needed here is not a tool that can run all unittests in > existence, but an official way for automated tools to run tests, with > the ability for any test and test framework to hook into that, so that > you can run any test suite automatically from an automated tool. The, > once that mechanism has been identified/implemented, we need to tell > everybody to do this. > > I don't care much what that mechanism is, but I think the easiest way > to get there is to tell people to extend distutils with a test command > (or use Distribute) and perhaps add such a command in 3.4 that will do > the unittest discover thingy. I remember looking into zope.testrunner > hooking into that mechanism as well, but I don't remember what the > outcome was. > > Doesn't setuptools/distribute already have a setup.py test command? That seems like the easiest way forward? > > //Lennart > ___ > Python-Dev mailing list > Python-Dev@python.org (mailto:Python-Dev@python.org) > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/donald.stufft%40gmail.com > > ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cffi in stdlib
A big +1 from me for cffi in the stdlib it's a great library. I just recently started using it to make bindings to a C library. I looked at the ctypes library, but haven't actually used it, because the docs confused me but with cffi I was able to get somewhere just by a liberal use of copy/paste from the C library's home page. The only C I know I figured out from working with cffi and fwiw even with that minimal amount of gained knowledge I still am confused by the Ctypes docs and where to even really start to try to use it. On Tuesday, February 26, 2013 at 10:13 AM, Maciej Fijalkowski wrote: > Hello. > > I would like to discuss on the language summit a potential inclusion > of cffi[1] into stdlib. This is a project Armin Rigo has been working > for a while, with some input from other developers. It seems that the > main reason why people would prefer ctypes over cffi these days is > "because it's included in stdlib", which is not generally the reason I > would like to hear. Our calls to not use C extensions and to use an > FFI instead has seen very limited success with ctypes and quite a lot > more since cffi got released. The API is fairly stable right now with > minor changes going in and it'll definitely stablize until Python 3.4 > release. Notable projects using it: > > * pypycore - gevent main loop ported to cffi > * pgsql2cffi > * sdl-cffi bindings > * tls-cffi bindings > * lxml-cffi port > * pyzmq > * cairo-cffi > * a bunch of others > > So relatively a lot given that the project is not even a year old (it > got 0.1 release in June). As per documentation, the advantages over > ctypes: > > * The goal is to call C code from Python. You should be able to do so > without learning a 3rd language: every alternative requires you to > learn their own language (Cython, SWIG) or API (ctypes). So we tried > to assume that you know Python and C and minimize the extra bits of > API that you need to learn. > > * Keep all the Python-related logic in Python so that you don’t need > to write much C code (unlike CPython native C extensions). > > * Work either at the level of the ABI (Application Binary Interface) > or the API (Application Programming Interface). Usually, C libraries > have a specified C API but often not an ABI (e.g. they may document a > “struct” as having at least these fields, but maybe more). (ctypes > works at the ABI level, whereas Cython and native C extensions work at > the API level.) > > * We try to be complete. For now some C99 constructs are not > supported, but all C89 should be, including macros (and including > macro “abuses”, which you can manually wrap in saner-looking C > functions). > > * We attempt to support both PyPy and CPython, with a reasonable path > for other Python implementations like IronPython and Jython. > > * Note that this project is not about embedding executable C code in > Python, unlike Weave. This is about calling existing C libraries from > Python. > > so among other things, making a cffi extension gives you the same > level of security as writing C (and unlike ctypes) and brings quite a > bit more flexibility (API vs ABI issue) that let's you wrap arbitrary > libraries, even those full of macros. > > Cheers, > fijal > > .. [1]: http://cffi.readthedocs.org/en/release-0.5/ > ___ > Python-Dev mailing list > Python-Dev@python.org (mailto:Python-Dev@python.org) > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/donald.stufft%40gmail.com > > ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
On Wednesday, February 20, 2013 at 6:22 PM, Antoine Pitrou wrote: > On Wed, 20 Feb 2013 18:21:22 -0500 > Donald Stufft mailto:donald.stu...@gmail.com)> > wrote: > > On Wednesday, February 20, 2013 at 6:08 PM, Antoine Pitrou wrote: > > > > It's not a distributed DoS issue, it's a severe DoS vulnerabilities. A > > > > single 1 kB XML document can kill virtually any machine, even servers > > > > with more than hundred GB RAM. > > > > > > > > > > > > > Assuming an attacker can inject arbitrary XML. Not every XML document > > > is loaded from the Internet. > > > > > > > > > Even documents not loaded from the internet can be at risk. Often times > > security breaches are the result of a chain of actions. You can say "I'm > > not loading this XML from the internet, so therefore I am safe" but then > > you have another flaw (for example) where you unpack a zip file > > without verifying there are not absolute paths and suddenly your xml file > > has > > been replaces with a malicious one. > > > > > Assuming your ZIP file is coming from the untrusted Internet, indeed. > Again, this is the same assumption that you are grabbing some important > data from someone you can't trust. > > No software you run on your computer grabs data from someone you don't trust and it all validates that even though you trust them they haven't been exploited? Like I said these sort of things are often caused by chaining several unrelated things together. > > Just because you are living in a Web-centric world doesn't mean > everyone does. There are a lot of use cases which are not impacted by > your security rules. Bugfix releases shouldn't break those use cases, > which means the security features should be mostly opt-in for 2.7 and > 3.3. > > Regards > > Antoine. > ___ > Python-Dev mailing list > Python-Dev@python.org (mailto:Python-Dev@python.org) > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/donald.stufft%40gmail.com > > ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
On Wednesday, February 20, 2013 at 6:23 PM, Christian Heimes wrote: > We can add a function to the XML package tree that enables all restrictions: > > * limit expansion depths of nested entities > * limit total amount of expanded chars > * disable external entity expansion > * optionally force expat to ignore and reset all DTD information > > 3rd party users have to disable secure settings explicitly for the > current interpreter (although expat limits are process wide and shared > across subinterpreters). > > try: > import xml.security > except ImportError: > # old Python > pass > else: > xml.security.harden_xml_parser() We've learned nothing from Ruby and their YAML problems. Things need to be safe by default and the unsafe things explicitly enabled. Even *smart* developers do things wrong (e.g. YAML.load instead of YAML.safe_load) and protecting developers by default should be the path forward. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
On Wednesday, February 20, 2013 at 6:08 PM, Antoine Pitrou wrote: > > It's not a distributed DoS issue, it's a severe DoS vulnerabilities. A > > single 1 kB XML document can kill virtually any machine, even servers > > with more than hundred GB RAM. > > > > > Assuming an attacker can inject arbitrary XML. Not every XML document > is loaded from the Internet. > > Even documents not loaded from the internet can be at risk. Often times security breaches are the result of a chain of actions. You can say "I'm not loading this XML from the internet, so therefore I am safe" but then you have another flaw (for example) where you unpack a zip file without verifying there are not absolute paths and suddenly your xml file has been replaces with a malicious one. > Not everyone is a security nuts. > > This is precisely why things should be safe by default and allow unsafe actions to be turned on optionally. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0
On Wednesday, February 20, 2013 at 2:48 AM, Chris Jerdonek wrote: > On Tue, Feb 19, 2013 at 3:16 PM, Daniel Holth (mailto:dho...@gmail.com)> wrote: > > Sorry, Chris must have meant http://hg.python.org/distlib/ . I was > > struggling to imagine a world where that is more visible than something on > > bitbucket. > > > > > I meant that bringing distlib into http://hg.python.org/cpython/ would > give it more visibility to core devs and others that already keep an > eye on python-checkins (the mailing list). And I think seeing the > Sphinx-processed docs integrated and cross-referenced with > http://docs.python.org/dev/ will help people understand better what > has been done and how it fits in with the rest of CPython -- which I > think would be useful to the community. It may also encourage > involvement (e.g. by being part of the main tracker). > > On the other hand it makes contributing to it more annoying since it does not have pull requests, unless it was just a mirror. > In asking about the "plan" for doing this, I was thinking of the > following remark by Nick: > > On Tue, Feb 19, 2013 at 5:40 AM, Nick Coghlan (mailto:ncogh...@gmail.com)> wrote: > > On Tue, Feb 19, 2013 at 11:23 PM, M.-A. Lemburg > (mailto:m...@egenix.com)> wrote: > > > > > > Hmm, what is distlib and where does it live ? > > > > As part of the post-mortem of packaging's removal from Python 3.3, > > several subcomponents were identified as stable and useful. distlib is > > those subcomponents extracted into a separate repository by Vinay > > Sajip. > > > > It will be proposed as the standard library infrastructure for > > building packaging related tools, while distutils will become purely a > > build system and have nothing to do with installing software directly > > (except perhaps on developer machines). > > > > > My question was basically whether there was a tentative plan for when > it (or completed parts of it) will be proposed (e.g. when a certain > amount of functionality is completed, etc). It's better not to do > this at the last minute if 3.4 is the plan (as I think was attempted > with packaging but for 3.3). > > On Tue, Feb 19, 2013 at 6:40 PM, Steven D'Aprano (mailto:st...@pearwood.info)> wrote: > > > > I keep hearing people say that the stdlib is not important, but I don't > > think > > that is true. There are lots of people who have problems with anything not > > in > > the standard library. > > > > - Beginners often have difficulty (due to inexperience, lack of confidence > > or > > knowledge) in *finding*, let alone installing and using, packages that > > aren't > > in the standard library. > > > > - To people in the Linux world, adding anything outside of your distro's > > packaging system is a nuisance. No matter how easy your packaging library > > makes it, you now have two sorts of packages: first-class packages that > > your distro will automatically update for you, and second-class ones that > > aren't. > > > > - People working in restrictive corporate systems often have to jump through > > flaming hoops before installing software. > > > > > I would also add that for people new to writing Python modules and > that want to distribute them, it's hard to evaluate what they are > "supposed" to use (distutils, setuptools, distribute, bento, etc). > Just a day or two ago, this exact question was asked on the Distutils > mailing list with subject "Confusion of a hobby programmer." Code not > being in the standard library creates an extra mental hurdle to > overcome. > > I agree that eventually the stdlib needs standard tooling to work with the future (™) but until that future is in use adding it to the stdlib feels premature to me. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0
On Tuesday, February 19, 2013 at 6:16 PM, Daniel Holth wrote: > Sorry, Chris must have meant http://hg.python.org/distlib/ . I was struggling > to imagine a world where that is more visible than something on bitbucket. > Half the comments have been about putting something in stdlib right away, > something that I do not understand at all as a voracious PyPI or > https://crate.io/ user. Let's not add anything to the stdlib till it has real world usage. Doing otherwise is putting the cart before the horse. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0
On Tuesday, February 19, 2013 at 3:25 PM, Paul Moore wrote: > On 19 February 2013 13:40, Nick Coghlan (mailto:ncogh...@gmail.com)> wrote: > > > If a tools wants to support metadata 2.0, it has to support all > > > the complicated stuff as well, i.e. handle the requires fields, > > > the environment markers and version comparisons/sorting. > > > > > > > > > Which is what distutils2 can be used for now, and what distlib will > > provide without the unwanted build system infrastructure in > > distutils2. > > > > > One particular item in Metadata 2.0 which needs to be addressed is > requirements specifiers, as used in (for example) Requires-Dist. These > take forms like "DistName (>= ver)". There are a number of potential > issues here: > > 1. This differs from the pkg_resources format (which omits the > parentheses). Having 2 subtly different formats is not a good idea in > the long term. At the moment, pkg_resources format is used in pip > requirements (on the command line and in requirement files) as well as > in setuptools/distribute and possibly elsewhere. > 2. There is currently no code that I am aware of that implements this > spec. I don't believe distlib does (yet - give Vinay 5 minutes and who > knows? :-)), pkg_resources as I said implements a different format, > and distutils2, apart from being a big dependency to cover just this > point, only matches the version (not the project name) and presumably > does so using the distutils2 version ordering (which is incompatible > with Metadata 2.0). > > It should be alongside the version stuff, atleast it was in distutils2. > 3. The format is fiddly to parse and process - nobody is likely to > implement it for themselves without a library that does it (especially > not when using pkg_resources is so easy). > > The PEP needs a rationale as to why the pkg_resources format wasn't > used, and should suggest a migration path for projects which currently > use the pkg_resources style (such as pip's requirements files). > > Paul > ___ > Python-Dev mailing list > Python-Dev@python.org (mailto:Python-Dev@python.org) > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/donald.stufft%40gmail.com > > ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] why do we allow this syntax?
On Friday, February 8, 2013 at 10:39 AM, Chris Withers wrote: > Hi All, > > Just had a bit of an embarrassing incident in some code where I did: > > sometotal =+ somevalue I'm guessing this gets parsed as sometotal = +somevalue > > I'm curious why this syntax is allowed? I'm sure there are good reasons, > but thought I'd ask... > > Chris > > -- > Simplistix - Content Management, Batch Processing & Python Consulting > - http://www.simplistix.co.uk > ___ > Python-Dev mailing list > Python-Dev@python.org (mailto:Python-Dev@python.org) > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/donald.stufft%40gmail.com > > ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Draft PEP for time zone support.
On Tuesday, December 11, 2012 at 3:31 PM, Barry Warsaw wrote: > On Dec 11, 2012, at 04:23 PM, Lennart Regebro wrote: > > > This PEP is also available on github: > > > > https://github.com/regebro/tz-pep/blob/master/pep-04tz.txt > > wget returns some html gobbledygook. Why-oh-why github?!' wget https://raw.github.com/regebro/tz-pep/master/pep-04tz.txt > > > PEP: 4?? > > I've assigned this PEP 431, reformatted a few extra wide paragraphs, committed > and pushed. > > Thanks Lennart! > -Barry > > ___ > Python-Dev mailing list > Python-Dev@python.org (mailto:Python-Dev@python.org) > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/donald.stufft%40gmail.com > > ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Conflicts [was Re: Keyword meanings [was: Accept just PEP-0426]]
On Saturday, December 8, 2012 at 9:11 PM, Steven D'Aprano wrote: > Why would a software package called "Spam" install a top-level module called > "Jam" rather than "Spam"? Isn't the whole point of Python packages to solve > this namespace problem? > Conflicts doesn't really solve file based conflicts as PJ Elby has pointed out tools need to detect that circumstance already. But to answer this question no, there is no required mapping between Project names (what your thing is called on PyPI) and python package names (what you import). Something named Spam on PyPI could provide multiple python packages, named whatever it was they wanted to be named. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Keyword meanings [was: Accept just PEP-0426]
On Friday, December 7, 2012 at 8:33 PM, Nick Coghlan wrote: > That's not what a Conflicts field is for. It's to allow a project to say > *they don't support* installing in parallel with another package. It doesn't > matter why it's unsupported, it's making a conflict perceived by the project > explicit in their metadata. > > Such a field is designed to convey information to users about *supported* > configurations, regardless of whether or not they happen to work for a given > use case. If a user believes a declared conflict is in error, and having the > two installed in parallel is important to them, they can: > 1. Use virtual environments to keep the two projects isolated from each other > 2. Use an installer that ignores Conflicts information (which will be all of > them, since that's the status quo) > 3. Make their case to the upstream project that the conflict has been > resolved, and installing the two in parallel no longer causes issues 4. Use the eventual --force flag that any installer that supported conflicts is likely to include ;) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Keyword meanings [was: Accept just PEP-0426]
On Thursday, December 6, 2012 at 6:28 AM, Vinay Sajip wrote: > Donald Stufft gmail.com (http://gmail.com)> writes: > > Never mind the "Obsoletes" information - even the more useful "Requires-Dist" > information is not exposed via PyPI, even though it appears to be stored in > the > database. (Or if it is, please point me to where - I must have missed it.) > > Requires-Dist doesn't exist for more than a handful of packages. But PyPI exposes it via the XMLRPC API, possibly the JSON api as well. > > Even if this were to be made available, it's presumably obtained from > PKG-INFO. > As I understand, this data is not considered reliable - for example, pip runs > egg_info on downloaded packages to get updated information when determining > dependencies to be downloaded. If the Requires-Dist info in PKG-INFO can't be > relied on, surely less critical information such as Obsoletes can't be relied > on, > either? > > pip runs egg_info because setuptools does not write out to PKG-INFO what the dependencies are (it does write it out to a different text file though). But IIRC that text file is not guaranteed to exist in the distribution. There's also the history where pip was trying to preserve as much backwards compat with easy_install as it could, and if you used the file that egg_info writes out then you'll only get the requirements for the system that the distribution was packaged on. Any if statements that affect the dependencies won't be in effect. > > Regards, > > Vinay Sajip > > ___ > Python-Dev mailing list > Python-Dev@python.org (mailto:Python-Dev@python.org) > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/donald.stufft%40gmail.com > > ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Keyword meanings [was: Accept just PEP-0426]
On Wednesday, December 5, 2012 at 6:18 PM, Barry Warsaw wrote: > On Dec 05, 2012, at 06:07 PM, Donald Stufft wrote: > > > If you're installing B you've prescribed trust to that author. If you don't > > trust the author then why are you installing (and then executing) code > > they wrote. > > > > > What you installed Z, but B got installed because it was a dependency three > levels down? > > Sure, you granted trust to Z, Z granted trust to Y, and Y granted trust to B. Like in SSL certificates there was a chain of trust. If you don't trust Z then don't install their package. > > > Very convenient to declare that one of the major use cases for > > Obsoletes over Obsoleted-By is not valid because of your own > > personal opinions. Like I said above, if you're installing a package > > that someone has uploaded you've implicitly granted them trust. There > > is far worse things that a bad Python citizen can do during, and after > > and install that what is allowed by Obsoletes. > > > > > Well, basically never installing anything from PyPI except into a virtualenv > is probably a good recommendation (maybe even now). > > A virtualenv only protects you from well behaved packages. There is no way to prevent a package author from doing very nasty things to you if they wish. Providing more power in the metadata doesn't make this situation better or worse, it just makes more standard paths in the cases where you do need to do it. > > > End systems often times do not have a singular organization controlling > > every package in their system. The best example is Ubuntu and their PPA's. > > > > > Well, PPAs are awesome, but have known and well-publicized trust issues. I > wouldn't enable a PPA into my running system without really knowing who the > owner is and why I'm using their PPA. Or doing a lot of testing in a chroot > first, and probably pinning the package set to just the one(s) from the PPA I > care about. > > Basically the same thing can be said about packages on PyPI. All the same trust issues exist there. Simply installing a Python package is already granting far more trust than Obsoletes requires since installing a package is executed someone else's python code on your system. Even if you remove setup.py you're still going to be executing their code on your system. If you do not trust the author of the packages you are installing, you do not install their packages. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Keyword meanings [was: Accept just PEP-0426]
On Wednesday, December 5, 2012 at 4:10 PM, PJ Eby wrote: > My point is that this can only work if the "obsoleting" is effectively > just a rename, in which case the field should be "renames", or better > still, "renamed-to" on the originating package. Arguing over Obsoletes vs Renames is a massive bikeshedding argument. > As I've mentioned repeatedly, Obsoleted-By handles more use cases than > Obsoletes, and has at least one practical automated use case > (notifying a developer that their project is depending on something > that's obsolete). > > Also, the example given as a use case in the PEP (Gorgon to Torqued) > is not just wrong, it's *actively misleading*. Gorgon and Torqued are > transparent renames of Medusa and Twisted, which do not share a common > API and thus cannot be used as the subject of any automated processing > (in the case of Obsoletes) without doing some kind of PyPI metadata > search for every package installed every time a package is installed. > > So it's a bad example. Hardly an argument against it. > 1. It cannot be used to prevent the installation of an obsolete > package without a PyPI metadata search, since you must examine every > *other* package on PyPI to find out whether some package obsoletes the > one you're trying to install. > > Will require support from PyPI but this ultimately isn't a big deal. > > 2. Unlike RPM, where metadata is provided by a trusted third party, > Obsoletes can be specified by any random forker (no pun intended), > which makes this information a mere advertisement... and an > advertisement to the wrong audience at that, because they must have > *already* found B in order to discover that it replaces A! > > If you're installing B you've prescribed trust to that author. If you don't trust the author then why are you installing (and then executing) code they wrote. > > 3. Nobody has yet supplied a use case where Obsoletes would not be > strictly improved upon by Obsoleted-By. (Note that "the author of > package X no longer maintains it" does not equal "package Y is > entitled to name itself the successor and enforce this upon all users" > -- this can work in RPM only because it is a third party Z who > declares Y the successor to X, and there is no such party Z in the > Python world.) > > Very convenient to declare that one of the major use cases for Obsoletes over Obsoleted-By is not valid because of your own personal opinions. Like I said above, if you're installing a package that someone has uploaded you've implicitly granted them trust. There is far worse things that a bad Python citizen can do during, and after and install that what is allowed by Obsoletes. > > > > I don't see this in this thread, could you link it again? > > http://mail.python.org/pipermail/catalog-sig/2010-October/003368.html > http://mail.python.org/pipermail/catalog-sig/2010-October/003364.html > > These posts also address why a "Conflicts" field is *also* unlikely to > be particularly useful in practice, in part for reasons that relate to > differences between RPM-land and Python-land. (For example, RPMs can > conflict over things besides files, due to runtime and configuration > issues that are out-of-scope for a Python installer tool.) > > I don't think Conflicts is something that every single package is going to require. As you said the tools themselves are going to handle the obvious cases for the bulk of situations. Unless you think there are no cases where two packages can conflict in more than what files are going to be installed then there are cases where it would be helpful and merely having the ability to use it when it is the best tool for the job isn't going to cause any great issue. > > While it's certainly desirable to not invent wheels, it's important to > understand that the Python community does not work the same way as a > Linux distribution. We are not a single organization shipping a > fully-functional and configured machine, we are hundreds of individual > authors shipping our own stuff. Conflict resolution and package > replacement (and even deciding what it is that things "provide" or > "require") are primarily *human* processes, not technical ones. > Relationship and support "contracts", IOW, rather than software > contracts. > > End systems often times do not have a singular organization controlling every package in their system. The best example is Ubuntu and their PPA's. > > Now, if the will of the community is to turn PyPI into a distro-style > repository, that's fine... but even if you completely ignore the human > issues, there are still technical ones. Generally, distro-style > repositories work by downloading the full metadata set (or at least an > index) to a user's machine. And that's the sort of architecture you'd > need in order for these type of fields to be technically feasible > (e.g., doing an index search for Obsoletes), without grinding the PyPI > servers into dust. This is insane. A fairly simple dat
Re: [Python-Dev] Keyword meanings [was: Accept just PEP-0426]
On Wednesday, December 5, 2012 at 2:13 AM, PJ Eby wrote: > On Mon, Dec 3, 2012 at 2:43 PM, Daniel Holth (mailto:dho...@gmail.com)> wrote: > > How to use Obsoletes: > > > > The author of B decides A is obsolete. > > > > A releases an empty version of itself that Requires: B > > > > B Obsoletes: A > > > > The package manager says "These packages are obsolete: A". Would you like to > > remove them? > > > > User says "OK". > > Um, no. Even if the the author of A and B are the same person, you > can't remove A if there are other things on the user's system using > it. The above scenario does not work *at all*, ever, except in the > case where B is simply an updated version of A (i.e. identical API) -- > in which case, why bother? To change the project name? (Then it > should be "Formerly-named" or something like that, not "Obsoletes".) > > You can automatically uninstall A from B in an automatic dependency management system. I *think* RPM does this, at the very least I believe it refuses to install B if A is already there (and the reverse as well).* There's nothing preventing an installer from, during it's attempt to install B, see it Obsoletes A, looking at what depends on A and warning the user what is going to happen and prompt it. I think Obsoletes as is an alright bit of information. I think the biggest flaw with Obsoletes isn't in Obsoletes itself, but is in the lack of a Conflicts tag that has the same functionality (minimally refusal to install both, possibly uninstall the previous one with a prompt to the user). Obsoletes has the semantics of a logical successor (typically renames) while Conflicts should have the semantics of a competitor. distribute would conflict with setuptools, foo2 would Obsoletes foo. * I could be wrong about RPM's treatment of Obsoletes > > Please, *please* see the previous Catalog-SIG discussion I linked: > this is only one of multiple metadata fields that were thoroughly > debunked in that discussion as completely useless for automated > dependency management. > > I don't see this in this thread, could you link it again? > ___ > Python-Dev mailing list > Python-Dev@python.org (mailto:Python-Dev@python.org) > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/donald.stufft%40gmail.com > > ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Accept just PEP-0426
On Tuesday, November 20, 2012 at 4:48 PM, PJ Eby wrote: > Words I agree that obsoletes is terrible, it's very specific and not something we particularly require. I'd much rather just have a generic "conflicts". ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Accept just PEP-0426
On Monday, November 19, 2012 at 9:24 PM, Daniel Holth wrote: > Mostly it seems a bit silly to have so much conversations about parts of the > pep that remain unchanged from previously accepted versions... Well, I think the PEP should describe what we expect to be implemented *shrug*. Either we should expect it to be implemented and it should be part of the spec, or we shouldn't expect people to implement it and it should be removed. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Accept just PEP-0426
On Monday, November 19, 2012 at 9:01 PM, Nick Coghlan wrote: > Look more closely at the docs for "Obsoletes" in RPM, not just those for > "Provides". Being able to transparently replace an existing package with a > renamed one that installs files with the same names is certainly part of the > purpose/capabilities of the RPM dependency machinery (i.e. precisely the > distribute vs setuptools situation). > We may want to clarify the wording to ensure it is clear that the provision > of the dist name (as posted on PyPI) is implied, though. > Cheers, > Nick. I'm not sure if you're responding to me or not, but we also have Obsoletes-Dist in the metadata. (Which I don't like the name of but that's for another argument). ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Accept just PEP-0426
On Monday, November 19, 2012 at 8:35 PM, Toshio Kuratomi wrote: > On Mon, Nov 19, 2012 at 07:49:41PM -0500, Donald Stufft wrote: > > Other languages seem to get along fine without it. Even the OS > > managers which have it don't allow it to be used to masquerade > > as another project, only to make generic virtual packages (e.g. "email"). > > > > I'm not sure this assertion about OS package managers is correct. I've only > just read: > http://www.python.org/dev/peps/pep-0426/#provides-dist-multiple-use > > but the rough rpm analogue seems to be the Provides: tag. > > Provides is given a string which is parsed into a name or a name and version > like this: > > Provides: python > Provides: python = 3.1.0 > > rpm has no way at package build time to tell that a particular name given in > a provides in one package is the actual name of another package. > > At installtime, rpm keeps package names and provides names separately but in > dependency comparisons either one can be used to satisfy a requirement. > What that means is that when asking about information on a package with name > "python", you'll get information about the python package with that name and > not about anything else that Provides: "python". But if you are installing > something that has a requirement on "python" either the package with the > name python or any package that Provides: python can satisfy the requirement. > Are you saying the RPM documentation is wrong? http://www.rpm.org/max-rpm/s1-rpm-inside-tags.html The provides tag is used to specify a *virtual package* that the packaged software makes available when it is installed. Normally, this tag would be used when different packages provide equivalent services. For example, any package that allows a user to read mail might provide the mail-reader virtual package. Another package that depends on a mail reader of some sort, could require the mail-reader virtual package. It would then install without dependency problems, if any one of several mail programs were installed. It pretty clearly states that it is not to be used for masquerading as a different package, which was my point. I wasn't making any claims about wether it was technically possible to do so or not, just what it's intended purpose was. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Accept just PEP-0426
So you want to leave metadata in that you think people shouldn't implement? Or you do think people should implement it and the point about it existing forever without an implementation is? At the very least there needs to be some sort of guidelines as to what to do with the field in the various states it could be in. On Monday, November 19, 2012 at 8:31 PM, Daniel Holth wrote: > We are getting along fine too. No tool parses metadata 1.x for package > management reasons and provides has existed forever with no implementation. > So it is not inconveniencing anyone. I would prefer to leave it alone. > > Daniel Holth > > On Nov 19, 2012, at 7:49 PM, Donald Stufft (mailto:donald.stu...@gmail.com)> wrote: > > > Other languages seem to get along fine without it. Even the OS > > managers which have it don't allow it to be used to masquerade > > as another project, only to make generic virtual packages (e.g. "email"). > > > > On Monday, November 19, 2012 at 7:43 PM, Daniel Holth wrote: > > > > > Does it really have baggage? I think it is necessary, although it doesn't > > > do favors to the implementer (and has never been implemented). How else > > > is anyone supposed to fork or merge projects? > > > > > > Daniel Holth > > > > > > On Nov 19, 2012, at 7:37 PM, PJ Eby > > (mailto:p...@telecommunity.com)> wrote: > > > > > > > On Mon, Nov 19, 2012 at 6:53 PM, Daniel Holth > > > (mailto:dho...@gmail.com)> wrote: > > > > > I think this PEP is a significant improvement from its predecessor. > > > > > It represents features like extras (provides-extra) and build > > > > > requirements (setup-requires-dist) that are in use in the Python > > > > > community but cannot be represented in older versions of the format, > > > > > it finally specifies a UTF-8 encoding, removes RFC 822, provides an > > > > > extension mechanism, and allows the description to be placed in the > > > > > document payload. > > > > > > > > Can we maybe kill Provides-Dist and its associated baggage first, > > > > though? > > > > > > > ___ > > > Python-Dev mailing list > > > Python-Dev@python.org (mailto:Python-Dev@python.org) > > > http://mail.python.org/mailman/listinfo/python-dev > > > Unsubscribe: > > > http://mail.python.org/mailman/options/python-dev/donald.stufft%40gmail.com > > > > > > > > > > > > > ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Accept just PEP-0426
On Monday, November 19, 2012 at 8:08 PM, Daniel Holth wrote: > The distro repos are centrally managed, too. Try getting setuptools to > provide a virtual package just because you want to fork.. and then update the > dependent packages? > > Daniel Holth Sorry didn't mean to make it sound like I thought we should do it like the OS packagers, just mentioning that very few (any?) other languages do it like that and they seem to be getting along just fine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Accept just PEP-0426
Other languages seem to get along fine without it. Even the OS managers which have it don't allow it to be used to masquerade as another project, only to make generic virtual packages (e.g. "email"). On Monday, November 19, 2012 at 7:43 PM, Daniel Holth wrote: > Does it really have baggage? I think it is necessary, although it doesn't do > favors to the implementer (and has never been implemented). How else is > anyone supposed to fork or merge projects? > > Daniel Holth > > On Nov 19, 2012, at 7:37 PM, PJ Eby (mailto:p...@telecommunity.com)> wrote: > > > On Mon, Nov 19, 2012 at 6:53 PM, Daniel Holth > (mailto:dho...@gmail.com)> wrote: > > > I think this PEP is a significant improvement from its predecessor. It > > > represents features like extras (provides-extra) and build requirements > > > (setup-requires-dist) that are in use in the Python community but cannot > > > be represented in older versions of the format, it finally specifies a > > > UTF-8 encoding, removes RFC 822, provides an extension mechanism, and > > > allows the description to be placed in the document payload. > > > > Can we maybe kill Provides-Dist and its associated baggage first, though? > > > ___ > Python-Dev mailing list > Python-Dev@python.org (mailto:Python-Dev@python.org) > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/donald.stufft%40gmail.com > > ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Accept just PEP-0426
On Monday, November 19, 2012 at 7:37 PM, PJ Eby wrote: > Can we maybe kill Provides-Dist and its associated baggage first, though? > > I would love to kill Provides-Dist. The biggest question there is how do you handle it's functionality? If someone needs setuptools but they have distribute installed they both shouldn't get installed. The need for it for the "2 packages being distributed together" I'm (personally) less concerned about since with proper dependency data we should be able to just depend on things instead of bundling them. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] performance of {} versus dict()
$ pypy -m timeit 'dict()'
10 loops, best of 3: 0.000811 usec per loop
$ pypy -m timeit '{}'
10 loops, best of 3: 0.000809 usec per loop
$ pypy -m timeit 'def md(**kw): return kw; md()'
1 loops, best of 3: 0.0182 usec per loop
$ pypy -m timeit -s 'def md(**kw): return kw' 'md()'
10 loops, best of 3: 0.00136 usec per loop
If the difference between dict() and {} is hurting your code why are
you still using CPython.
On Wednesday, November 14, 2012 at 5:51 PM, MRAB wrote:
> On 2012-11-14 21:40, Greg Ewing wrote:
> > Chris Angelico wrote:
> > > Perhaps an alternative question: What can be done to make the latter
> > > less unpalatable?
> > >
> >
> >
> > * We could introduce a new syntax such as {a = 1, b = 2}.
> >
> > * If the compiler were allowed to recognise builtins, it could
> > turn dict(a = 1, b = 2) into {'a':1, 'b':2} automatically.
> >
>
> That would be a transformation of the AST, although it assumes that
> 'dict' hasn't been rebound.
>
> Should there be the option of a warning if a builtin is rebound? Or the
> option of the transformation plus a warning if the builtin is rebound?
> ___
> Python-Dev mailing list
> Python-Dev@python.org (mailto:Python-Dev@python.org)
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/donald.stufft%40gmail.com
>
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Distutils] [Catalog-sig] accept the wheel PEPs425, 426, 427
Distutils is not good enough for the vast majority of people. Even for what it does, it does not do it well. It is a library that is user hostile and buggy. It was a fine first revision of packaging, but the Python community needs something better. On Tuesday, November 13, 2012 at 11:31 AM, a.cava...@cavallinux.eu wrote: > > +1 > Distutils is good enough: it could be better but for what is required > (essentially copying files and creating packages installers) is fine. The only > corner case is an absolute pain in the neck is in the cross compile scenario. > > Currently I don't have *any* need for "auto" tools (setuptools and > descendants). > As an example I always advice to avoid like a plague anything that depends on > setuptools... with all the due respect I think is the poor's developer > attempt to > play the sys admin game. > > Even virtual env is a poor work around on the python interpreter not being > relocatable (as in a portable app fashion). > > thanks > > > On Tue 13/11/12 16:10, "Ronald Oussoren" ronaldousso...@mac.com > (mailto:ronaldousso...@mac.com) wrote: > > > > On 13 Nov, 2012, at 16:00, Daniel Holth > .com> wrote: > > > > > I want to remove distutils from the standard > > library. > > Why? Distutils may not be perfect, but is usable for basic packages. It > > could even be enhanced to support these peps and be even more useable, > > although patches for that would run into the self-imposed freeze of > > distutils development. > > Ronald > > ___ > > Distutils-SIG maillist - distutils-...@python.org > > (mailto:distutils-...@python.org) > > http://mail.python.org/mailman/listinfo/distutils-sig > > > > > > ___ > Distutils-SIG maillist - distutils-...@python.org > (mailto:distutils-...@python.org) > http://mail.python.org/mailman/listinfo/distutils-sig > > ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Edits to Metadata 1.2 to add extras (optional dependencies)
On Friday, September 14, 2012 at 12:30 PM, Daniel Holth wrote: > Description: is the only multi-line field in the metadata. It is > almost never needed at runtime. It would be great for performance and > simplify the parser to just put it in another file. License also can be multiline I believe, ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Edits to Metadata 1.2 to add extras (optional dependencies)
On Friday, September 14, 2012 at 12:30 PM, Daniel Holth wrote: > Add to metadata 1.3: > > Description-File: README(\..+)? > > Meaning the description should be read from a file in the same > directory as PKG-INFO or METADATA (including in the .dist-info > directories) and we strongly recommend it be named as README.* and be > utf-8 encoded text. > > Description: is the only multi-line field in the metadata. It is > almost never needed at runtime. It would be great for performance and > simplify the parser to just put it in another file. > > Mutually exclusive with Description. > > May beg for a Summary: tag with a one-line description. > ___ > Python-Dev mailing list > Python-Dev@python.org (mailto:Python-Dev@python.org) > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/donald.stufft%40gmail.com > > -1 on this personally ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] packaging location ?
On Thursday, September 13, 2012 at 7:32 AM, Tarek Ziadé wrote: > > Yeah but we've been too ambitious. > > Here's my proposal - actually it's Nick's proposal but I want to make > sure we're on the same > page wrt steps, and I think that addresses Antoine concerns > > 1. create a new package, called pkglib (or whatever), located at hg > .python.org (http://python.org) as a new project that just strictly contains : > > - the PEP implementations > - non controversial features like files parser, pypi index browser etc > > it's doable - since that's what we have done in distutils2. the > modules that implements those PEPs are standalone > > Let's avoid by all means to put the old distutils command logic there. > > Let's have a strict process on every new thing we're adding there. > > > 2. make pkglib python 2 *and* python 3 compatible - natively, not w/ 2to3 > > > 3. make distutils2, distribute, pip, bento, etc. use that and try to > share as many bits as possible > > > 4. ask each project to pour in pkglib anything that can be reused by others I started messing around with yanking some of the parts of distutils2 (things I've been calling packaging primitives for lack of a better word). Don't have anything particularly usable yet, but the approach you're talking about is similar to what I started to do. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] packaging location ?
On Thursday, September 13, 2012 at 5:38 AM, Antoine Pitrou wrote: > Most people use distutils / packaging as > an application, not a library. If you provide only a subset of > the necessary features, people won't use packaging. Not that I think current usage patterns matter since moving from setup.py to a static file, but that's not really true. The wide proliferation of setuptools shows pretty clearly that people are fine using distutils as a library. Even beyond that the popularity of pip shows that as well since very few people even directly interact with setup.py at all except to create the distributions. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Edits to Metadata 1.2 to add extras (optional ependencies)
On Friday, August 31, 2012 at 6:48 AM, "Martin v. Löwis" wrote: > 3. There should be a specification of how collisions between extension > fields and standard fields are resolved. E.g. if I have > > Extension: Home > Home-page: http://www.python.org > > is Home-page the extension field or the PEP 345 field? There are > several ways to resolve this; I suggest giving precedence to the > standard field (unless you specify that extensions must follow all > standard fields, in which case you can drop the extension prefix > from the extension keys). > Unless i'm mistaken (which I may be!) I believe that a / can be used as the separator between the namespace and the "real" key. Home-page: http://www.python.org Extension: Home Home/other-thing: Foo Doing this is the "Extension" field required?___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Edits to Metadata 1.2 to add extras (optional dependencies)
On Tuesday, August 28, 2012 at 11:07 AM, "Martin v. Löwis" wrote: > > What happens when it expires? Is that name freed up for future use? > > > Yes, exactly. > > > I > > think that freeing up the name is likely to be a bad idea since we can't go > > backwards in time (as you alluded to later about not deleting them), so > > what does expiration do? > > > > > Why would it require going backwards in time? Existing usages of the > extension just become invalid, e.g. with the consequence that you can't > upload the package to PyPI anymore unless you remove the extension, > or re-register it. > > If the extension is in active use, somebody certainly will make sure it > stays registered. Expiration is to free up names that are not in active > use, but are otherwise reasonable names for metadata fields (say, > Requires-Unicode-Version). What do you do with packages that have already been uploaded with requires-unicode-version once it expires? If the point of a registry is to remove ambiguity from what any particular key means, won't expiring and allowing reregistration of an in use name (even if it's no longer being uploaded, but is still available inside of a package) reintroduce that same ambiguity? How will we know that requires-unicode-version from a package uploaded a year ago and has since expired is different than requires-unicode-version from a package uploaded yesterday and has been reregistered? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Edits to Metadata 1.2 to add extras (optional dependencies)
On Tuesday, August 28, 2012 at 11:05 AM, Nick Coghlan wrote: > On Wed, Aug 29, 2012 at 12:53 AM, Donald Stufft (mailto:donald.stu...@gmail.com)> wrote: > > Please, don't. The software and infrastructure to run PyPI exists. > Some level of namespacing makes sense to separate out extension > management to different groups of people, but creating a whole > management application just for this would be serious overkill. > How do you deal with a PyPI package foo which wants a bar-test value (foo-bar-test), and a PyPI package foo-bar with a value test (foo-bar-test). PyPI packages allow too much in the way of names to be able to fully namespace it without collisions. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Edits to Metadata 1.2 to add extras (optional dependencies)
On Tuesday, August 28, 2012 at 10:43 AM, "Martin v. Löwis" wrote: > > I'm happy for PyPI to host such a registry. A specificaion for the > registry should be part of the PEP for the 1.3 format, but I would > propose this structure (without having researched in detail what > other registries feature, but with a rough idea what IANA registries > typically include): PyPI packages itself could serve as a registry, but I like the idea of a separate registry better in many ways because it lets you divorce the namespace from the package. The question being would this be a x-registered-name type system or a registered-namespace-* type system? It occurs to me one problem with arbitrary namespaces is there is a unintended collision problem. e.g. you have the foo-bar namespace and the foo namespace, what happens if you have a test key inside of foo-bar and a bar-test inside of the foo namepspace. They'll both end up being foo-bar-test. This makes me think that we need a seperate registry and that if we go the namespace route it should be limited to alphanumerics only so that you don't have the foo/foo-bar collision problem. > > - name of metadata field > - name of registrant (individual or PyPI package) > - contact email address (published) > - expiration date; by default, extensions expire 1 month after > their registration, unless renewed; maximum expiration time is > 5 years > - English description of the field > - regular expression to validate the field What happens when it expires? Is that name freed up for future use? I think that freeing up the name is likely to be a bad idea since we can't go backwards in time (as you alluded to later about not deleting them), so what does expiration do? > > Deleting undesired extensions would not be possible, instead, one > would have to create another extension if the syntax or semantics > changes ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Edits to Metadata 1.2 to add extras (optional dependencies)
On Tuesday, August 28, 2012 at 9:41 AM, Nick Coghlan wrote: > > The only thing I really care about is the namespacing, for the same > reasons the IETF wrote RFC 6648, as Petri linked earlier [1]. > Establishing proper name registration rules can categorically > eliminate a bunch of problems further down the line (such as the past > confusion between which metadata entries were defined by PEPs and > which were setuptools-specific extensions that other tools might not > understand). > > I'm happy with any form of a namespace to be quite honest. I have a bit of a preference for no or flat namespace but i'm perfectly fine with a PyPI based namespace. The important part is a defined way to extend the data that even when tools don't understand the extended data they can losslessly move it around from setup.cfg/setup.py/whatever to METADATA and any other format, even if they themselves don't utilize it, leaving it intact for tools that _do_ utilize it. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Edits to Metadata 1.2 to add extras (optional dependencies)
On Tuesday, August 28, 2012 at 9:09 AM, Nick Coghlan wrote: > > It does have the advantage that tools for manipulating the format can > remain dumber, but that doesn't seem like *that* much of an advantage, > especially since any such benefit could be eliminated completely by > just switching to a completely standard ConfigParser format by putting > the PEP defined settings into a [python] section. > To be more specific, there is setup.cfg (which I dislike for other reasons), and then there is METADATA. setup.cfg is an ini file but METADATA is a simple key: value file with a flat namespace so any namespacing you want to do in METADATA needs to be done at the key level. You could translate: [setuptools] requires-dist=foo in a setup.cfg into setuptools-requires-dist: foo in METADATA, but I'm not sure if that would be beneficial or not. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Edits to Metadata 1.2 to add extras (optional dependencies)
On Tuesday, August 28, 2012 at 9:09 AM, Nick Coghlan wrote: > On Tue, Aug 28, 2012 at 10:57 PM, Daniel Holth (mailto:dho...@gmail.com)> wrote: > > How about > > > > Extensions are fields that start with a pypi-registered name followed > > by a hyphen. A file that contains extension fields declares them with > > Extension: name : > > > > Extension: pypiname > > pypiname-Field: value > > > > > The repetition seems rather annoying. Compare the two section based > variants I just posted to: > > Extension: wheel > wheel-Version: 0.9 > wheel-Packager: bdist_wheel-0.1 > wheel-Root-Is-Purelib: true > > It does have the advantage that tools for manipulating the format can > remain dumber, but that doesn't seem like *that* much of an advantage, > especially since any such benefit could be eliminated completely by > just switching to a completely standard ConfigParser format by putting > the PEP defined settings into a [python] section. > METADATA files are not ini files. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Edits to Metadata 1.2 to add extras (optional dependencies)
On Tuesday, August 28, 2012 at 8:28 AM, Nick Coghlan wrote: > > Agreed, and this is the kind of thing a v1.3 metadata PEP could > define. It just needs to be properly namespaced, and the obvious > namespacing mechanism is PyPI project names. The biggest reason I have against namespacing them is it makes moving from experimental to standard easier, but I'm ok with some form of a namespace. The biggest reason I see against using PyPI names as the namespace is it needlessly ties a piece of data to the original creator. Similar to how right now you could write a less hacky setuptools, but in order to do so you need to continue to use the setuptools package name (see distribute). Using PyPI names means that in the requires-dist example it would be something like setuptools-requires-dist, and even if I make my own tool that supports the same concept as setuptools's requires-dist I would need to use setuptools-requires-dist. The concept of metadata I think should be divorced from specific implementations. Obviously there are going to be some implementation specific issues but I think it's much cleaner to have a x-requires-dist that any implementation can use than to have whoever-invented-it-first-requires-dist or a twenty-different-forms-of-requires-dist. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Edits to Metadata 1.2 to add extras (optional dependencies)
I personally think that at a minimum we should have X-Fields that get moved into the normal METADATA file, and personally I would prefer to just drop the X- prefix completely. I think any spec which doesn't include first class support for extending it with new metadata is going to essentially kick the can down the road and solve the problems of today without leaving room to solve the problems of tomorrow. I know that distutils2 have requires-dist, but for the sake of argument pretend they don't. If there is first class support for extending the metadata with new fields, a project could come along, and add a requires-dist (or x-requires-dist) concept to metadata. Tools that understand it would see that data and be able to act on it, tools that don't understand it would simply write it to the METADATA file incase in the future a tool that does understand it needs to act on it. Essentially first class support for extending the metadata outside of a PEP process means that outside of the stdlib people can experiment and try new things, existing tools will continue to work and just ignore that extra data (but leave it intact), new tools will be able to utilize it to do something useful. Ideally as a new concept is tested externally and begins to gain acceptance a new metadata version could be created that standardizes that field as part of the spec instead of an extension. On Tuesday, August 28, 2012 at 7:45 AM, Nick Coghlan wrote: > On Tue, Aug 28, 2012 at 9:04 PM, Daniel Holth (mailto:dho...@gmail.com)> wrote: > > Setuptools just uses path.exists() when it needs a particular file and will > > not bother parsing pkg-info at all if it can help it. The metadata edits > > for 1.2 fold some of those files into metadata. > > > You can't use path.exists() on metadata published by a webservice (or > still inside a zipfile), but you can download or read the main > metadata file. > > Still, I don't really care whether or not such a field indicating the > presence of custom metadata is added, I'm mainly registering a strong > -1 on allowing extension fields (in the form of X- headers or CSS > style prefixed headers) in the metadata file itself. > > Cheers, > Nick. > > -- > Nick Coghlan | ncogh...@gmail.com (mailto:ncogh...@gmail.com) | Brisbane, > Australia > ___ > Python-Dev mailing list > Python-Dev@python.org (mailto:Python-Dev@python.org) > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/donald.stufft%40gmail.com > > ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of packaging in 3.3
On Friday, June 22, 2012 at 4:55 PM, Terry Reedy wrote: > > Every time windows users download and install a binary, they are taking > a chance. I try to use a bit more sense than some people, but I know it > is not risk free. There *is* a third party site that builds installers, > but should I trust it? I would prefer that (except perhaps for known and > trusted authors) PyPI compile binaries, perhaps after running code > through a security checker, followed by running it through one or more > virus checkers. > I think you overestimate the abilities of "security checkers" and antivirus. Installing from PyPI is a risk, wether you use source or binaries. There is currently not a very good security story for installing python packages from PyPI (not all of this falls on PyPI), but even if we get to a point there is, PyPI can never be as safe as installing from RPM's or DEB and somewhat mores in the case of binaries. You _have_ to make a case by case choice if you trust the authors/maintainers of a particular package. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signed packages
Not at the moment, but I could gather them up and make them public later today. They are very rough draft at the moment. On Friday, June 22, 2012 at 1:09 PM, Alexandre Zani wrote: > On Fri, Jun 22, 2012 at 9:56 AM, Donald Stufft (mailto:donald.stu...@gmail.com)> wrote: > > On Friday, June 22, 2012 at 12:54 PM, Alexandre Zani wrote: > > > > > > Key distribution is the real issue though. If there isn't a key > > distribution infrastructure in place, we might as well not bother with > > signatures. PyPI could issue x509 certs to packagers. You wouldn't be > > able to verify that the name given is accurate, but you would be able > > to verify that all packages with the same listed author are actually > > by that author. > > > > I've been sketching out ideas for key distribution, but it's very much > > a chicken and egg problem, very few people sign their packages (because > > nothing uses it currently), and nobody is motivated to work on > > infrastructure > > or tooling because no one signs their packages. > > > > > Are those ideas available publicly? I would love to chip in. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signed packages
On Friday, June 22, 2012 at 12:54 PM, Alexandre Zani wrote: > > Key distribution is the real issue though. If there isn't a key > distribution infrastructure in place, we might as well not bother with > signatures. PyPI could issue x509 certs to packagers. You wouldn't be > able to verify that the name given is accurate, but you would be able > to verify that all packages with the same listed author are actually > by that author. > > I've been sketching out ideas for key distribution, but it's very much a chicken and egg problem, very few people sign their packages (because nothing uses it currently), and nobody is motivated to work on infrastructure or tooling because no one signs their packages. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signed packages
Ideally authors will be signing their packages (using gpg keys). Of course how to distribute keys is an exercise left to the reader. On Friday, June 22, 2012 at 11:48 AM, Vinay Sajip wrote: > v.loewis.de (http://v.loewis.de)> writes: > > > > > See above. Also notice that such signing is already implemented, as part > > of PEP 381. > > > > > BTW, I notice that the certificate for https://pypi.python.org/ expired a week > ago ... > > Regards, > > Vinay Sajip > > > ___ > Python-Dev mailing list > Python-Dev@python.org (mailto:Python-Dev@python.org) > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/donald.stufft%40gmail.com > > ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of packaging in 3.3
On Friday, June 22, 2012 at 6:20 AM, David Cournapeau wrote: > If by manifest you mean the build manifest, then that's not desirable: > the manifest contains the explicit filenames, and those are > platform/environment specific. You don't want this to be user-facing. > It appears I misunderstood the files that bento uses then ;) It is late (well early now) and I have not used bento extensively. What I suggest mirrors RPM's similarly except the build step (when there is indeed a build step) is handled by a python script included in the package by the author of said package. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of packaging in 3.3
On Friday, June 22, 2012 at 5:52 AM, Dag Sverre Seljebotn wrote: > > The reason PyPI isn't one big security risk is that packages are built > from source, and so you can have some confidence that backdoors would be > noticed and highlighted by somebody. > > Having a common standards for binary installation phase would be great > sure, but security-minded users would still need to build from source in > every case (or trust a 3rt party build farm that builds from source). > The reason you can trust RPMs at all is because they're built from SRPMs. > > Dag The reason you trust RPM's is not because they are built from SRPM's, but because you trust the people running the repositories. In the case of PyPI you can't make a global call to implicitly trust all packages because there is no gatekeeper as in an RPM system, so it falls to the individual to decide for him or herself which authors they trust and which authors they do not trust. But this proposal alludes to both source dists and built dists, either which may be published and installed from. In the case of a source dist the package format would include all the metadata of the package. Included in that is a python script that knows how to build this particular package (if special steps are required). This script could simply call out to an already existing build system, or if simple enough work on it's own. Source dists would also obviously contain the source. In the case of a binary dist the package format would include all the metadata of the package, plus the binary files. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of packaging in 3.3
On Friday, June 22, 2012 at 5:22 AM, Dag Sverre Seljebotn wrote: > > What Bento does is have one metadata file for the source-package, and > another metadata file (manifest) for the built-package. The latter is > normally generated by the build process (but follows a standard > nevertheless). Then that manifest is used for installation (through > several available methods). > > >From what I understand, this dist.(yml|json|ini) would be replacing the mainfest not the bento.info then. When bento builds a package compatible with the proposed format it would instead of generating it's own manifest it would generate the dist.(yml|json|ini). ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of packaging in 3.3
I think json probably makes the most sense, it's already part of the stdlib for 2.6+ and while it has some issues with human editablity, there's no reason why this json file couldn't be auto generated from another data structure by the "package creation tool" that exists outside of the stdlib (or inside, but outside the scope of this proposal). Which is really part of what I like a lot about this proposal, how you come about the final product doesn't matter, distutils, bento, yet-uncreated-tool, manually crafting tar balls and files, you could describe your data in yaml, python, or going towards more magical ends of things, it could be automatically generated from your filesystem. It doesn't matter, all that matters is you create your final archive with the agreed upon structure and the agreed upon dist.(yml|json|ini) and any compliant installer should be able to install it. On Friday, June 22, 2012 at 3:56 AM, Vinay Sajip wrote: > Nick Coghlan gmail.com (http://gmail.com)> writes: > > > ini-style is often good enough, and failing that there's json. Or, you > > just depend on PyYAML :) > > > > > Except when PyYAML is packaged and distributed using dist.yaml :-) > > Regards, > > Vinay Sajip > > ___ > Python-Dev mailing list > Python-Dev@python.org (mailto:Python-Dev@python.org) > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/donald.stufft%40gmail.com > > ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of packaging in 3.3
On Friday, June 22, 2012 at 1:05 AM, Nick Coghlan wrote: > > - I reject setup.cfg, as I believe ini-style configuration files are > not appropriate for a metadata format that needs to include file > listings and code fragments > > - I reject bento.info (http://bento.info), as I think if we accept > yet-another-custom-configuration-file-format into the standard library > instead of just using YAML, we're even crazier than is already > apparent > > - I shall use "dist.yaml" as my proposed name for my "I wish I could > define packages like this" format (and yes, that means adding yaml > support to the standard library is part of the wish) > > - many of the details below will be flawed, but I want to give a clear > idea for how a concept like this might work in practice > > - we need to define a clear set of build phases, and then design the > dist metadata format accordingly. For example: > - source > - uses a "source" section in dist.yaml > - "source/install" maps source files directly to desired > install locations > - essentially what the setup.cfg Resources section tries to do > - used for pure Python code, documentation, etc > - See below for example > - "source/files" defines a list of extra files to be included > - "source/exclude" defines the list of files to be excluded > - "source/run" defines a Python fragment to be executed > - serves a similar purpose to the "files" section in setup.cfg > - creates a temporary directory (and sets it as the working directory) > - dist.yaml is copied to the temporary directory > - all files to be installed are copied to the temporary directory > - all extra files are copied to the temporary directory > - the Python fragment in "source/run" is executed (which can > thus easily add more files) > - if sdist archive creation is requested, entire contents of > temporary directory are included > - build > - uses a "build" section in dist.yaml > - "build/install" maps built files to desired install locations > - like source/install, but for build artifacts > - compiled C extensions, .pyc and .pyo files, etc would all go here > - "build/run" defines a Python fragment to be executed > - "build/files" defines the list of files to be included > - "build/exclude" defines the list of files to be excluded > - "build/requires" defines extra dependencies not needed at runtime > - starting environment is a source directory that is either: > - preexisting (e.g. to allow building in-place in the source tree) > - created by running source first > - created by unpacking an sdist archive > - the Python fragment in "build/run" is executed to trigger the build > - if the build succeeds (i.e. doesn't throw an exception) > - create a temporary directory > - copy dist.yaml > - copy all specified files > - this is the easiest way to exclude build artifacts from > the distribution, while still keeping them around to enable > incremental builds > - if bdist_simple archive creation is requested, entire > contents of temporary directory are included > - other bdist formats (such as bdist_rpm) will have their own > rules for getting from the bdist_simple format to the platform > specific format > - install > - uses an "install" section in dist.yaml > - "install/pre" defines a Python fragment to be executed > before copying files > - "install/post" defines a Python fragment to be executed > after copying files > - starting environment is a bdist_simple directory that is either: > - preexisting (e.g. to allow creation by system packaging tools) > - created by running build first > - created by unpacking a bdist_simple archive > - end result is a fully installed and usable piece of software > - test > - uses a "test" section in dist.yaml > - "test/run" defines a Python fragment to be executed to start the tests > - "test/requires" defines extra dependencies needed to run the > test suite > I dislike some of the (implementation) details, but in general I think this is a good direction to go in. Less trying to force tools to work together by hijacking setup.py or something and more "this is a package, it contains the data you need to install, and how to install it, you installation tool can use this data however it pleases to make sure it is installed." I feel like this is (one of?) the missing piece of the puzzle to define a set of standards that _any_ package creation, or installation tool can implement and gain interoperability. I don't want to argue over implementation details as I think that is premature right now, so this concept has a big +1 from me. RPM, deb, etc has a long history and a lot of shared knowledge so looking at them and adapting it to work cross platform is likely to be huge win. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of packaging in 3.3
On Thursday, June 21, 2012 at 7:34 PM, Alex Clark wrote: > Hi, > > On 6/21/12 5:38 PM, Donald Stufft wrote: > > On Thursday, June 21, 2012 at 4:01 PM, Paul Moore wrote: > > > End users should not need packaging tools on their machines. > > > > Sort of riffing on this idea, I cannot seem to find a specification for > > what a Python > > package actually is. > > > > > > FWIW according to distutils[1], a package is: a module or modules inside > another module[2]. So e.g.:: > > > foo.py is a module > > > and: > > foo/__init__.py > foo/foo.py > > is a simple package containing the following modules: > > import foo, foo.foo > > > Alex > > > [1] > http://docs.python.org/distutils/introduction.html#general-python-terminology > > [2] And a distribution is a compressed archive of a package, in case > that's not clear. > Right, i'm actually talking about distributions. (As is everyone else in this thread). And that a definition is not a specification. What i'm trying to get at is with a standard package format where all the metadata is able to get gotten at without the packaging lib (distutils/setuptools cannot get at metadata without using distutils or setuptools). It would need to be required that this serves as the one true source of metadata and that other tools can add certain types of metadata to this format. If say distutils2 wrote a package that adhered to a certain standard, and wrote all the information that distutils2 knows about how to install said package (what files, names, versions, dependencies etc) to a file (say PKG-INFO) that contained only "common" standard information then another tool (say bento) could take that package, and install it. The idea i'm hoping for is to stop worrying about one implementation over another and hoping to create a common format that all the tools can agree upon and create/install. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of packaging in 3.3
On Thursday, June 21, 2012 at 4:01 PM, Paul Moore wrote: > End users should not need packaging tools on their machines. > Sort of riffing on this idea, I cannot seem to find a specification for what a Python package actually is. Maybe the first effort should focus on this instead of arguing one implementation or another. As a packager: I should not (in general) care what tool (pip, pysetup, easy_install, buildout, whatever) is used to install my package, My package should just describe what to do to install itself. As a end user: I should not (in general) care what tool was used to create a package (setuptools, bento, distutils, whatever). My tool of choice should look at the package and preform the operations that the package says are needed for install. Ideally the package could have some basic primitives that are enough to tell the package installer tool what to do to install it, These primitives should be enough to cover the common cases (pure python modules at the very least, maybe additionally some C modules). Now as others have remarked it would be insane to attempt to do this in every case as it would involve writing a build system that is more advanced than anything else existing, so a required primitive would be something that allows calling out to a specific package decided build system (waf, make, whatever) to handle the build configuration. The eventual end goal here being to make a package from something that varies from implementation to implementation to a standardized format that any number of tools can build on top of. It would likely include some things defining where metadata MUST be defined. For instance, if metadata in setuptools was "compiled" down to static file, and easy_install, pip et;al used that static file to install from instead of executing setup.py, then the end user would not have required setup tools installed and instead any number of tools could have been created that utilized that data. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of packaging in 3.3
On Wednesday, June 20, 2012 at 2:36 AM, Victor Stinner wrote: > What is the status of the third party module on PyPI (distutils2)? > Does it contain all fixes done in the packaging module? Does it have > exactly the same API? Does it support Python 2.5 to 3.3, or maybe also > 2.4? > > How is the distutils2 module installed? Installed manually? Using pip > or setuptools? Is distutils2 included in some Linux distributions? > > If it's simple to install distutils2, it's not a big deal to not have > it in the stdlib. > > -- > > It is sometimes a pain to have a buggy module in Python. For example, > I got a lot of issues with the subprocess module of Python 2.5. I > started to include a copy of the subprocess module from Python 2.7 in > my projects to workaround these issues. > > In my previous work we did also backport various modules to get the > last version of the xmlrpc client on Python 2.5 (especially for > HTTP/1.1, to not open a new TCP socket at each request). > > I don't want to reopen the discussion "the stdlib should be an > external project". I just want to confirm that it is better to wait > until important users of the packaging API have finished their work > (on porting their project to distutils2, especially pip), before we > can declare the module (and its API) as stable. > > By the way, what is the status of "pip using distutils2"? Some students started on a pip2 that was based on distutils2, but I don't think they've really done much/anything with actually using distutils2 and have mostly been working on other parts. > > Victor > ___ > Python-Dev mailing list > Python-Dev@python.org (mailto:Python-Dev@python.org) > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/donald.stufft%40gmail.com > > ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives
On Tuesday, March 13, 2012 at 9:31 AM, Paul Moore wrote: > On 13 March 2012 03:48, C. Titus Brown mailto:c...@msu.edu)> > wrote: > > I feel like there's a middle ground where stable, long-term go-to modules > > could > > be mentioned, though. I don't spend a lot of time browsing PyPI, but I > > suspect > > almost everyone spends a certain amount of time in the Python docs (which > > is a > > testimony to their quality IMO). So I'm in favor of conservative link-outs > > but without any deprecating language. > > > > > I applaud the idea of promoting the many excellent packages available. > It can be very hard to separate the good from the indifferent (or even > bad) when browsing PyPI. I've found some very good packages recently > which I'd never have known about without some random comment on a > mailing list. > > However, I'm not keen on having the stdlib documentation suggest that > I should be using something else. No code should ever be documenting > "don't use me, there are better alternatives" unless it is deprecated > or obsolete. > > On the other hand, I would love to see a community-maintained document > that described packages that are acknowledged as "best of breed". That > applies whether or not those packages replace something in the stdlib. > Things like pywin32, lxml, and requests would be examples in my > experience. There's no reason this *has* to be in the core > documentation - it may be relevant that nothing has sprung up > independently yet... > > http://python-guide.org ? > > Maybe a separate item in the Python documentation, "External Modules", > could be created and maintained by the community? By being in the > documentation, it has a level of "official recommendation" status, and > by being a top-level document it's visible (more so than, for example, > a HOWTO document would be). Because it's in the released > documentation, it is relatively stable, which implies that external > modules would need to have a genuine track record to get in there, but > because it's community maintained it should reflect a wider consensus > than just the core developers' views. > > Paul. > ___ > Python-Dev mailing list > Python-Dev@python.org (mailto:Python-Dev@python.org) > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/donald.stufft%40gmail.com > > ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Counting collisions for the win
I believe that either solution has the potential to break existing applications so to ensure that no applications are broken there will need to be a method of disabling it. I also believe that to maintain the backwards compatibility that Python has traditionally had in bug fix releases that either solution will need to default to off. Given those 2 things that I believe, I don't think that the argument should be which solution will break less, because I believe both will need to be off by default, but which solution more adequately solves the underlying problem. On Friday, January 20, 2012 at 5:11 PM, Terry Reedy wrote: > On 1/20/2012 2:51 PM, Donald Stufft wrote: > > > I think the counting collision is at best a bandaid and not a proper fix > > stemmed from a desire to not break existing applications on a bugfix > > release ... > > > > > My opinion of counting is better than yours, but even conceding the > theoretical, purity argument, our release process is practical as well. > There have been a few occasions when fixes to bugs in our code have been > delayed from a bugfix release to the next feature release -- because the > fix would break too much code depending on the bug. > > Some years ago there was a proposal that we should deliberately tweak > hash() to break 'buggy' code that depended on it not changing. This > never happened. So it has been left de facto constant, to the extent it > is, for some years. > > -- > Terry Jan Reedy > > ___ > Python-Dev mailing list > Python-Dev@python.org (mailto:Python-Dev@python.org) > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/donald.stufft%40gmail.com > > ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Counting collisions for the win
On Friday, January 20, 2012 at 2:36 PM, Tres Seaver wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > On 01/20/2012 02:04 PM, Donald Stufft wrote: > > > Even if a MemoryException is raised I believe that is still a > > fundamental change in the documented contract of dictionary API. > > > > > How so? Dictionary inserts can *already* raise that error. Because it's raising it for a fundamentally different thing. "You have plenty of memory, but we decided to add an arbitrary limit that has nothing to do with memory and pretend you are out of memory anyways". > > > I don't believe there is a way to fix this without breaking someones > > application. The major differences I see between the two solutions is > > that counting will break people's applications who are otherwise > > following the documented api contract of dictionaries, > > > > > Do you have a case in mind where legitimate user data (not crafted as > part of a DoS attack) would trip the 1000-collision limit? How likely is > it that such cases exist in already-deployed applications, compared to > the known breakage in existing applications due to hash randomization? > > I don't, but as there's never been a limit on how many collisions a dictionary can have, this would be a fundamental change in the documented (and undocumented) abilities of a dictionary. Dictionary key order has never been guaranteed, is documented to not be relied on, already changes depending on if you are using 32bit, 64bit, Jython, PyPy etc or as someone else pointed out, to any number of possible improvements to dict. The counting solution violates the existing contract in order to serve people who themselves are violating the contract. Even with their violation the method that I +1'd still serves to not break existing applications by default. > > > and randomization will break people's applications who are violating > > the documented api contract of dictionaries. > > > > Personally I feel that the lesser of two evils is to reward those who > > followed the documentation, and not reward those who didn't. > > > > > Except that I think your set is purely hypothetical, while the second set > is *lots* of deployed applications. > > Which is why I believe that it should be off by default on the bugfix, but easily enabled. (Flag, env var, whatever). That allows people to upgrade to a bugfix without breaking their application, and if this vulnerability affects them, they can enable it. I think the counting collision is at best a bandaid and not a proper fix stemmed from a desire to not break existing applications on a bugfix release which can be better solved by implementing the real fix and allowing people to control (only on the bugfix, on 3.3+ it should be forced to on always) if they have it enabled or not. > > > Tres. > - -- > === > Tres Seaver +1 540-429-0999 tsea...@palladion.com > (mailto:tsea...@palladion.com) > Palladion Software "Excellence by Design" http://palladion.com > -BEGIN PGP SIGNATURE- > Version: GnuPG v1.4.10 (GNU/Linux) > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ > > iEYEARECAAYFAk8ZwlgACgkQ+gerLs4ltQ4KOACglAHDgn5wUb+cye99JbeW0rZo > 5oAAn2ja7K4moFLN/aD4ZP7m+8WnwhcA > =u7Mt > -END PGP SIGNATURE- > > ___ > Python-Dev mailing list > Python-Dev@python.org (mailto:Python-Dev@python.org) > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/donald.stufft%40gmail.com > > ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Counting collisions for the win
Even if a MemoryException is raised I believe that is still a fundamental change in the documented contract of dictionary API. I don't believe there is a way to fix this without breaking someones application. The major differences I see between the two solutions is that counting will break people's applications who are otherwise following the documented api contract of dictionaries, and randomization will break people's applications who are violating the documented api contract of dictionaries. Personally I feel that the lesser of two evils is to reward those who followed the documentation, and not reward those who didn't. So +1 for Randomization as the only option in 3.3, and off by default with a flag or environment variable in bug fixes. I think it's the only way to proceed that won't hurt people who have followed the documented behavior. On Friday, January 20, 2012 at 1:49 PM, Brett Cannon wrote: > > > On Fri, Jan 20, 2012 at 13:15, Guido van Rossum (mailto:gu...@python.org)> wrote: > > On Fri, Jan 20, 2012 at 5:10 AM, Barry Warsaw > (mailto:ba...@python.org)> wrote: > > > On Jan 20, 2012, at 01:50 PM, Victor Stinner wrote: > > > > > > >Counting collision doesn't solve this case, but it doesn't make the > > > >situation worse than before. Raising quickly an exception is better > > > >than stalling for minutes, even if I agree than it is not the best > > > >behaviour. > > > > > > ISTM that adding the possibility of raising a new exception on dictionary > > > insertion is *more* backward incompatible than changing dictionary order, > > > which for a very long time has been known to not be guaranteed. You're > > > running some application, you upgrade Python because you apply all > > > security > > > fixes, and suddenly you're starting to get exceptions in places you can't > > > really do anything about. Yet those exceptions are now part of the > > > documented > > > public API for dictionaries. This is asking for trouble. Bugs will > > > suddenly > > > start appearing in that application's tracker and they will seem to the > > > application developer like Python just added a new public API in a > > > security > > > release. > > > > Dict insertion can already raise an exception: MemoryError. I think we > > should be safe if the new exception also derives from BaseException. We > > should actually eriously consider just raising MemoryException, since > > introducing a new built-in exception in a bugfix release is also very > > questionable: code explicitly catching or raising it would not work on > > previous bugfix releases of the same feature release. > > > > > OTOH, if you change dictionary order and *that* breaks the application, > > > then > > > the bugs submitted to the application's tracker will be legitimate bugs > > > that > > > have to be fixed even if nothing else changed. > > > > There are lots of things that are undefined according to the language spec > > (and quite possibly known to vary between versions or platforms or > > implementations like PyPy or Jython) but which we would never change in a > > bugfix release. > > > > > So I still think we should ditch the paranoia about dictionary order > > > changing, > > > and fix this without counting. A little bit of paranoia could creep back > > > in > > > by disabling the hash fix by default in stable releases, but I think it > > > would > > > be fine to make that a compile-time option. > > > > I'm sorry, but I don't want to break a user's app with a bugfix release and > > say "haha your code was already broken you just didn't know it". > > > > Sure, the dict order already varies across Python implementations, possibly > > across 32/64 bits or operating systems. But many organizations (I know a > > few :-) have a very large installed software base, created over many years > > by many people with varying skills, that is kept working in part by very > > carefully keeping the environment as constant as possible. This means that > > the target environment is much more predictable than it is for the typical > > piece of open source software. > > > > Sure, a good Python developer doesn't write apps or tests that depend on > > dict order. But time and again we see that not everybody writes perfect > > code every time. Especially users writing "in-house" apps (as opposed to > > frameworks shared as open source) are less likely to always use the most > > robust, portable algorithms in existence, because they may know with much > > more certainty that their code will never be used on certain combinations > > of platforms. For example, I rarely think about whether code I write might > > not work on IronPython or Jython, or even CPython on Windows. And if > > something I wrote suddenly needs to be ported to one of those, well, that's > > considered a port and I'll just accept that it might mean changing a few > > things. > > > > The time to break a dependency on dict order is not with a b
Re: [Python-Dev] [PATCH] Adding braces to __future__
I don't always post to python-dev, but when I do I ask for braces. On Friday, December 9, 2011 at 4:43 PM, Antoine Pitrou wrote: > > Dear Cedric, > > I'm guessing you drank too much (perhaps you are training for New Year's > Eve), ate some bad sausages or are simply very self-complacent. > python-dev is not the place where to post long unstructured ramblings > with no practical value. Consider writing on your personal blog > instead. > > Thank you > > Antoine. > > > > On Fri, 9 Dec 2011 21:26:29 +0100 > Cedric Sodhi mailto:man...@gmx.net)> wrote: > > IF YOU THINK YOU MUST REPLY SOMETHING WITTY, ITERATE THAT THIS HAD BEEN > > DISCUSSED BEFORE, REPLY THAT "IT'S SIMPLY NOT GO'NNA HAPPEN", THAT "WHO > > DOESN'T LIKE IT IS FREE TO CHOOSE ANOTHER LANGUAGE" OR SOMETHING > > SIMILAR, JUST DON'T. > > > > Otherwise, read on. > > > > I know very well that this topic has been discussed before. On forums. > > Mailing lists. IRC. Blogs. From person to person, even. > > > > And I know equally well, from all those years experiencing > > argument-turned-debates on the internet, how a (minor|major) fraction of > > participants make up for their inability to lead a proper debate by > > speaking the loudest of all, so that eventually quantity triumphs over > > quality and logic. > > > > That ahead; I hope you can try not to fall in that category. Let instead > > reason prevail over sentimentalism, mislead purism, elitism, and all > > other sorts of isms which hinder advancement in the greater context. > > > > Python has surprised once already: The changes from 2 to 3 were not > > downwards compatible because the core developers realized there is more > > to a sustainable language than constantly patching it up until it comes > > apart like the roman empire. > > > > Let's keep that spirit for a second and let us discuss braces, again, > > with the clear goal of improving the language. > > > > End of disclaimer? > > > > End of disclaimer! > > > > Whitespace-Blocking (WSB) as opposed to Delimiter-Blocking (DB) has > > reasons. What are those reasons? Well, primarily, it forces the > > programmer to maintain well readable code. Then, some might argue, it is > > quicker to type. > > > > Two reasons, but of what importance are they? And are they actually > > reasons? > > > > You may guessed it from the questions themselves that I'm about to > > question that. > > > > I don't intend to connote brazen implications, so let me spell out what > > I just implied: I think anyone who thinks that exclusive WSB is a good > > alternative or even preferable to DB is actually deluding themselves for > > some personal version of one of those isms mentioned above. > > > > Let's examine these alleged advantages objectively one for one. But > > before that, just to calm troubled waters a little, allow me bring > > forward the conclusion: > > > > Absolutely no intentions to remowe WSB from Python. Although one might > > have gotten that impression from the early paragraphs, no intentions to > > break downwards compatibility, either. > > > > What Python needs is an alternative to WSB and can stay Python by still > > offering WSB to all those who happen to like it. > > > > Readable code, is it really an advantage? > > > > Two linebreaks, just for the suspense, then: > > > > Of course it is. > > > > Forcing the programmer to write readable code, is that an advantage? No > > suspense, the answer is Of course not. > > > > Python may have started off as the casual scripting language for casual > > people. People, who may not even have known programming. And perhaps it > > has made sense to force -- or shall we say motivate, since you can still > > produce perfectly obfuscated code with Python -- them to write readably. > > > > But Python has matured and so has its clientele. Python does not become > > a better language, neither for beginners nor for experienced programmers > > who also frequently use Python these days, by patronizing them and > > restricting them in their freedom. > > > > Readable code? Yes. Forcing people to write readable code by artificial > > means? No. > > > > Practice is evidence for the mischief of this policy: Does the FOSS > > community suffer from a notorious lack of proper indention or > > readability of code? Of course we don't. > > > > I'm not a native speaker, but dict.cc (http://dict.cc) tells me that what > > we call "mit > > Kanonen auf Spatzen schießen" (firing cannons at sparrows) is called > > breaking a fly on the wheel in English. > > > > I may lack the analogy for the fly on the wheel, which, if I'm not > > mistaken, used to be a device for torture in the Middle Ages, but I can > > tell you that the cannon ball which might have struck the sparrows, > > coincidently caused havoc in the hinterlands. > > > > For the wide-spread and professional language Python is today, the idea > > of forcing people to indent is misguided. These days, it may address
