Re: [Python-Dev] Moving Python 2.7 [was: 3.5] on Windows to a new compiler
Brian Curtin writes: > Adding features into 3.x is already not enough of a carrot on the > stick for many users. Intentionally leaving 2.7 on a dead compiler is > like beating them with the stick. No, it's like a New Year's resolution to stop self-flagellating, and handing the whip to the users to use on themselves, or not, as they choose. Remember, the users *chose* to remain locked-in to 2.7, hoping that we would continue to provide support, maybe 2.8. They had alternatives: contributing resources (in full-time developer support units!) to the PSF earmarked for Python 2, porting their dependencies to Python 3, etc. All expensive, yes, but eventually they need to pay the price of support or switching. Staying with Python 2 was always a bet that switching would be cheaper in the future, or that they'd have more resources in the future, or both. Who knows about the private resources, but not only does Python 3 acquire more features steadily, but efforts in core by folks like Ethan, distutils, and Nick (just to name those I've followed personally), along with steadily and expanding ports of 3rd party libraries, are quickly making switching cheaper. Cheap *enough*? That's for the users themselves to decide. So I'm not arguing against support; this kind of support (*and* the people who argue that it's worth doing, and then *do* it!) is one reason I have *no* hesitation in recommending Python (3!) vs. any comparable language.[1] But whatever is decided here, we're doing it for pride or for our own use, not because we owe the users anything. Footnotes: [1] I don't know enough about languages like Ruby or Perl to say Python provides strictly better support. I just can't imagine that it gets better than this! ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
On 7 June 2014 01:41, Steve Dower wrote: > > What this means for Python is that C extensions for Python 3.5 and later can > be built using any version of MSVC from 14.0 and later. Those who are aware > of the current state of affairs where you need to use a matching compiler > will hopefully see how big an improvement this will be. It is also likely > that other compilers will have an easier time providing compatibility with > this new CRT, making it simpler and more reliable to build extensions with > LLVM or GCC against an MSVC CPython. \o/ That's great news. (I'm assuming that change in policy includes figuring out a solution to the file descriptor problem, since we determined during the Stackless 2.8 discussion that file descriptor mismatches were actually our biggest stumbling block when it came to mixing and matching different CRT versions in one process) > Basically, what I am offering to do is: > > * Update the files in PCBuild to work with Visual Studio "14" > * Make any code changes necessary to build with VC14 > * Regularly test the latest Python source with the latest MSVC builds and > report issues/suggestions to the MSVC team > * Keep all changes in a separate (public) repo until early next year when > we're getting close to the final VS "14" release > > What I am asking anyone else to do is: > > * Nothing > > Thoughts/comments/concerns? As long as we're also keeping the VS10 files up to date as a fallback option, which we will be, since the VS14 work will be in a separate repo, this sounds like a fine idea to me. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
On 7 June 2014 15:05, Donald Stufft wrote: > I don’t particularly care too much though, I just think that bumping > the compiler in a 2.7.Z release is a really bad idea and that either > of the other two options are massively better. It is *incredibly* unlikely that backwards compatibility with binary extensions will be broken within the Python 2.7 series - there's a reason we said "No" when the Stackless folks were asking about it a while back. Instead, the toolchain availability problem is currently being tackled by trying to make suitable build toolchains more readily available (both the official VS 2008 toolchain and alternative open source toolchains), and by reducing the reliance on building from source for end users. Both of those courses of action are likely to bear fruit. It's only in the case where those approaches *don't* solve the problem that we'll need to come back and revisit the question of a compatibility break for binary extensions - it is, as you say, a really bad idea, and hence not something we would pursue when there are better options available (I think a Python 2.8 release would be an *even worse* idea in terms of souring our relationships with redistributors, but fortunately, those aren't our only two choices). Regards, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
On Jun 7, 2014, at 12:58 AM, Nick Coghlan wrote: > On 7 June 2014 14:47, Donald Stufft wrote: >> On Jun 7, 2014, at 12:41 AM, Nick Coghlan wrote: >>> >>> Words like "just", or "simple", or "easy" really have no place being >>> applied to a task where the time required to fully execute it with *no >>> significant problems* is still measured in years. >> >> How much of that time exists because there were actual significant >> changes from 2.6 to 2.7 and how much of it would not need to exist >> if 2.8 was literally 2.7.Z with a new compiler on Windows. IOW is it >> the *version* number that causes the slow upgrade, or is it the fact >> that there are enough changes that it can’t be safely applied >> automatically. > > It's the version number change itself. Python 2.7 was covered by the > language moratorium, so it consists almost entirely of standard > library changes, and the porting notes are minimal: > https://docs.python.org/2/whatsnew/2.7.html#porting-to-python-2-7 I’m not sure I agree, the porting docs only show a subset of changes, you also have a lot of new stuff like OrderedDict, dict comprehensions, set literals, argparse, dict views, memory views, etc. AFAIK stable releases don’t jump versions because all of these new features are risks, not because a number didn’t change. I don’t particularly care too much though, I just think that bumping the compiler in a 2.7.Z release is a really bad idea and that either of the other two options are massively better. > > We didn't even switch compilers on Windows (both 2.6 and 2.7 use VS 2008). > > I can't think of a better demonstration than the slow pace of the > Python 2.7 rollout that the challenges with doing a new minor release > of Python really aren't technical ones at the language level - they're > technical and administrative challenges in the way the language > version number interacts with the broader Python ecosystem, especially > the various redistribution channels. > > Cheers, > Nick. > > -- > Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
On 7 June 2014 14:47, Donald Stufft wrote: > On Jun 7, 2014, at 12:41 AM, Nick Coghlan wrote: >> >> Words like "just", or "simple", or "easy" really have no place being >> applied to a task where the time required to fully execute it with *no >> significant problems* is still measured in years. > > How much of that time exists because there were actual significant > changes from 2.6 to 2.7 and how much of it would not need to exist > if 2.8 was literally 2.7.Z with a new compiler on Windows. IOW is it > the *version* number that causes the slow upgrade, or is it the fact > that there are enough changes that it can’t be safely applied > automatically. It's the version number change itself. Python 2.7 was covered by the language moratorium, so it consists almost entirely of standard library changes, and the porting notes are minimal: https://docs.python.org/2/whatsnew/2.7.html#porting-to-python-2-7 We didn't even switch compilers on Windows (both 2.6 and 2.7 use VS 2008). I can't think of a better demonstration than the slow pace of the Python 2.7 rollout that the challenges with doing a new minor release of Python really aren't technical ones at the language level - they're technical and administrative challenges in the way the language version number interacts with the broader Python ecosystem, especially the various redistribution channels. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
On 7 June 2014 14:01, Chris Barker wrote: >> >> Why not just define Python 2.8 as Python 2.7 except with a newer compiler? >> I cannot see why that would be massive undertaking, if changing compiler >> for 2.7 is neccesary anyway. > > > A reminder that this was brought up a few months ago, as a proposal by the > stackless team, as they wanted to use a newer compiler for binaries. IIRC, > there was a pretty resounding "don't do that" from this list. Makes sense to > me -- we have how many different binaries of 2.7 on how many platforms, with > how many compilers? Sure, python.org has been nicely consistent about what > compiler (run time, really) they use to distribute Windows binaries, but the > python version has NOTHING to do with what compiler is used. (for hat matter > there is 32 bit and 64 bit 2.7 on Windows ...) Supported by python-dev? We have two: 32-bit and 64-bit, both depending on the Microsoft C runtime, and both published as binary installers on python.org. That's it. > I think, at the time, it was thought that pip, wheel, and the metadata > standards should be extended to allow multiple binaries of the same version > with different compilers to be in the wild. those projects have had bigger > fish to fry, but maybe it's time to get ahead of the game with that, so we > can accommodate this change. It's already getting hard to find VS2008 > Express, and building 64 bit extensions is s serious pain. That was a largely independent discussion, noting that if we come up with a mechanism for dealing with Linux distro variances, it may also be useful for dealing with Windows C runtime variances. Regards, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
On Jun 7, 2014, at 12:41 AM, Nick Coghlan wrote: > On 7 June 2014 08:43, Sturla Molden wrote: >> Brett Cannon wrote: >> >>> Nope. A new minor release of Python is a massive undertaking which is why >>> we have saved ourselves the hassle of doing a Python 2.8 or not giving a >>> clear signal as to when Python 2.x will end as a language. >> >> Why not just define Python 2.8 as Python 2.7 except with a newer compiler? >> I cannot see why that would be massive undertaking, if changing compiler >> for 2.7 is neccesary anyway. > > It's honestly astonishing the number of people that tell us doing a > new minor release of Python 2 is easy, and then refuse to believe us > when we tell them it isn't. > > It's 2014 and Python *2.7*, which was released in *2010*, is STILL > BEING ROLLED OUT. One part of the rollout that is near & dear to my > own heart is the fact that Red Hat Enterprise Linux 7 and CentOS 7 are > still in their respective release candidate phases, and it is the 6 -> > 7 transition that finally upgrades their system Pythons from 2.6 to > 2.7. Maya 2014 & MotionBuilder 2014 are also the first versions > Autodesk are shipping that use 2.7 rather than 2.6 as the scripting > engine (although my understanding is that Autodesk don't guarantee > compatibility with Python C extensions that aren't built specifically > for use with their products, so they already use a newer C runtime on > Windows than we do). > > And once those two dominoes fall, then there'll be some additional > follow on upgrade work in some parts of the developer community as the > *users* that receive their Python through those channels rather than > directly from upstream switch from 2.6 to 2.7 and stumble over the > small compatibility breaks between those two releases. > > Words like "just", or "simple", or "easy" really have no place being > applied to a task where the time required to fully execute it with *no > significant problems* is still measured in years. How much of that time exists because there were actual significant changes from 2.6 to 2.7 and how much of it would not need to exist if 2.8 was literally 2.7.Z with a new compiler on Windows. IOW is it the *version* number that causes the slow upgrade, or is it the fact that there are enough changes that it can’t be safely applied automatically. > > That said, there are definitely problems with toolchain availability > on Windows for Python 2, and it isn't clear yet how that will be > addressed in the long run. Steve is working on ensuring the official > toolchain and C runtime binaries are more readily available from MS. > Other folks are independently looking into ensuring that open source > toolchains (like mingw) can be used effectively to at least build > Python C extensions for Windows (and ironing out some of the glitches > with that approach that others have mentioned). The Python Packaging > Authority are continuing to work on the wheel based infrastructure to > help avoid end users having to compile anything in the first place, > and redistributors like ActiveState, Enthought & Continuum Analytics > also make it possible for many end users to just ignore these upstream > concerns. An extension compatibility break would be an absolutely last > resort, pursued only if all other attempts at resolving the challenges > have demonstrably failed - even at the best of times it can take > months for C extension authors to start publishing compatible binaries > for a new minor release, so we'd have to assume that time would be > even longer for a Python 2.7 maintenance release, if they published > updated binaries at all. > > Regards, > Nick. > > -- > Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/donald%40stufft.io - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
On 7 June 2014 08:43, Sturla Molden wrote: > Brett Cannon wrote: > >> Nope. A new minor release of Python is a massive undertaking which is why >> we have saved ourselves the hassle of doing a Python 2.8 or not giving a >> clear signal as to when Python 2.x will end as a language. > > Why not just define Python 2.8 as Python 2.7 except with a newer compiler? > I cannot see why that would be massive undertaking, if changing compiler > for 2.7 is neccesary anyway. It's honestly astonishing the number of people that tell us doing a new minor release of Python 2 is easy, and then refuse to believe us when we tell them it isn't. It's 2014 and Python *2.7*, which was released in *2010*, is STILL BEING ROLLED OUT. One part of the rollout that is near & dear to my own heart is the fact that Red Hat Enterprise Linux 7 and CentOS 7 are still in their respective release candidate phases, and it is the 6 -> 7 transition that finally upgrades their system Pythons from 2.6 to 2.7. Maya 2014 & MotionBuilder 2014 are also the first versions Autodesk are shipping that use 2.7 rather than 2.6 as the scripting engine (although my understanding is that Autodesk don't guarantee compatibility with Python C extensions that aren't built specifically for use with their products, so they already use a newer C runtime on Windows than we do). And once those two dominoes fall, then there'll be some additional follow on upgrade work in some parts of the developer community as the *users* that receive their Python through those channels rather than directly from upstream switch from 2.6 to 2.7 and stumble over the small compatibility breaks between those two releases. Words like "just", or "simple", or "easy" really have no place being applied to a task where the time required to fully execute it with *no significant problems* is still measured in years. That said, there are definitely problems with toolchain availability on Windows for Python 2, and it isn't clear yet how that will be addressed in the long run. Steve is working on ensuring the official toolchain and C runtime binaries are more readily available from MS. Other folks are independently looking into ensuring that open source toolchains (like mingw) can be used effectively to at least build Python C extensions for Windows (and ironing out some of the glitches with that approach that others have mentioned). The Python Packaging Authority are continuing to work on the wheel based infrastructure to help avoid end users having to compile anything in the first place, and redistributors like ActiveState, Enthought & Continuum Analytics also make it possible for many end users to just ignore these upstream concerns. An extension compatibility break would be an absolutely last resort, pursued only if all other attempts at resolving the challenges have demonstrably failed - even at the best of times it can take months for C extension authors to start publishing compatible binaries for a new minor release, so we'd have to assume that time would be even longer for a Python 2.7 maintenance release, if they published updated binaries at all. Regards, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
> > > Why not just define Python 2.8 as Python 2.7 except with a newer compiler? > I cannot see why that would be massive undertaking, if changing compiler > for 2.7 is neccesary anyway. > A reminder that this was brought up a few months ago, as a proposal by the stackless team, as they wanted to use a newer compiler for binaries. IIRC, there was a pretty resounding "don't do that" from this list. Makes sense to me -- we have how many different binaries of 2.7 on how many platforms, with how many compilers? Sure, python.org has been nicely consistent about what compiler (run time, really) they use to distribute Windows binaries, but the python version has NOTHING to do with what compiler is used. (for hat matter there is 32 bit and 64 bit 2.7 on Windows ...) I think, at the time, it was thought that pip, wheel, and the metadata standards should be extended to allow multiple binaries of the same version with different compilers to be in the wild. those projects have had bigger fish to fry, but maybe it's time to get ahead of the game with that, so we can accommodate this change. It's already getting hard to find VS2008 Express, and building 64 bit extensions is s serious pain. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
On 6/6/2014 9:13 PM, Donald Stufft wrote: On Jun 6, 2014, at 9:05 PM, Terry Reedy wrote: If you are suggesting that a Windows compiler change should be invisible to non-Windows users, I agree. Let us assume that /pcbuild remains for those who have vc2008 and that /pcbuild14 is added (and everything else remains as is). Then the only other thing that would change is the Windows installer released on Python.org. Call than 2.7.9W or whatever on the download site and interactive startup message to signal that something is different. How are packaging tools supposed to cope with this? AFAIK there is nothing in most of them to deal with a X.Y.Z release suddenly dealing with a different compiler. For this option, packaging tools on Windows would have to gain a special rule to cope with a special, hopefully unique, not-to-be-repeated, series of releases. If VC2008 ceases to become available to those who do not already have it, and who machines do not break or get replaced, dealing with a different easily available compiler would be easier than dealing with having no compiler. -- Terry Jan Reedy ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [numpy wishlist] Interpreter support for temporary elision in third-party classes
Nathaniel Smith wrote: >> with numpy.accelerate: >> x = >> y = >> z = >> # evaluation of x,y,z happens here > > Using an alternative evaluation engine is indeed another way to > optimize execution, which is why projects like numexpr, numba, theano, > etc. exist. But this is basically switching to a different language in > a different VM. I was not thinking that complicated. Let us focus on what an unmodified CPython can do. A compound expression with arrays can also be seen as a pipeline. Imagine what would happen if in "NumPy 2.0" arithmetic operators returned coroutines instead of temporary arrays. That way an expression could be evaluated chunkwise, and the chunks would be small enough to fit in cache. Sturla ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [numpy wishlist] Interpreter support for temporary elision in third-party classes
Greg Ewing wrote: > Julian Taylor wrote: >> tp_can_elide receives two objects and returns one of three values: >> * can work inplace, operation is associative >> * can work inplace but not associative >> * cannot work inplace > > Does it really need to be that complicated? Isn't it > sufficient just to ask the object potentially being > overwritten whether it's okay to overwrite it? How can it know this without help from the interpreter? Sturla ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
Brian Curtin wrote: >> If Python 2.7 users are left with a dead compiler on Windows, they will >> find a solution. For example, Enthought is already bundling their Python >> distribution with gcc 2.8.1 on Windows. > > Again, not something I think we should depend on. A lot of people use > python.org installers. I am not talking about changing the python.org installers. Let it remain on VS2008 for Python 2.7. I am only suggesting we make it easier to find a free C compiler compatible with the python.org installers. The NumPy/SciPy dev team have taken the burden to build a MinGW toolchain that is configured to be 100 % ABI compatible with the python.org installer. I am only suggesting a link to it or something like that, perhaps even host it as a separate download. (It is GPL, so anyone can do that.) That way it would be easy to find a compatible C compiler. We have to consider that VS2008 will be unobtainable abandonware long before the promised Python 2.7 support expires. When that happens, users of Python 2.7 will need to find another compiler to build C extensions. If Python.org makes this easier it would hurt less to have Python 2.7 remain on VS2008 forever. Sturla ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
On Jun 6, 2014, at 9:05 PM, Terry Reedy wrote: > On 6/6/2014 6:47 PM, Nathaniel Smith wrote: >> On Fri, Jun 6, 2014 at 11:43 PM, Sturla Molden >> wrote: >>> Brett Cannon wrote: >>> Nope. A new minor release of Python is a massive undertaking which is why we have saved ourselves the hassle of doing a Python 2.8 or not giving a clear signal as to when Python 2.x will end as a language. >>> >>> Why not just define Python 2.8 as Python 2.7 except with a newer compiler? >>> I cannot see why that would be massive undertaking, if changing compiler >>> for 2.7 is neccesary anyway. >> >> This would require recompiling all packages on OS X and Linux, even >> though nothing had changed. > > If you are suggesting that a Windows compiler change should be invisible to > non-Windows users, I agree. > > Let us assume that /pcbuild remains for those who have vc2008 and that > /pcbuild14 is added (and everything else remains as is). Then the only other > thing that would change is the Windows installer released on Python.org. Call > than 2.7.9W or whatever on the download site and interactive startup message > to signal that something is different. > > -- > Terry Jan Reedy > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/donald%40stufft.io How are packaging tools supposed to cope with this? AFAIK there is nothing in most of them to deal with a X.Y.Z release suddenly dealing with a different compiler. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
On Jun 6, 2014 6:33 PM, "Sturla Molden" wrote: > > Brian Curtin wrote: > > > Well we're certainly not going to assume such a thing. I know people do > > that, but many don't (I never have). > > If Python 2.7 users are left with a dead compiler on Windows, they will > find a solution. For example, Enthought is already bundling their Python > distribution with gcc 2.8.1 on Windows. Again, not something I think we should depend on. A lot of people use python.org installers. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
On 6/6/2014 6:47 PM, Nathaniel Smith wrote: On Fri, Jun 6, 2014 at 11:43 PM, Sturla Molden wrote: Brett Cannon wrote: Nope. A new minor release of Python is a massive undertaking which is why we have saved ourselves the hassle of doing a Python 2.8 or not giving a clear signal as to when Python 2.x will end as a language. Why not just define Python 2.8 as Python 2.7 except with a newer compiler? I cannot see why that would be massive undertaking, if changing compiler for 2.7 is neccesary anyway. This would require recompiling all packages on OS X and Linux, even though nothing had changed. If you are suggesting that a Windows compiler change should be invisible to non-Windows users, I agree. Let us assume that /pcbuild remains for those who have vc2008 and that /pcbuild14 is added (and everything else remains as is). Then the only other thing that would change is the Windows installer released on Python.org. Call than 2.7.9W or whatever on the download site and interactive startup message to signal that something is different. -- Terry Jan Reedy ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Internal representation of strings and Micropython
On 7 Jun 2014 00:53, "Paul Sokolovsky" wrote: > > Yes. Except for one small detail - Python3 specifies these code points > to be Unicode code points. And Unicode is a very bloated thing. I rather suspect users of East Asian & African scripts might have a different notion of what constitutes "bloated" vs "can actually represent this language properly, unlike 8-bit code spaces". > But if we drop that "Unicode" stipulation, then it's also exactly what > MicroPython implements. Its "str" type consists of codepoints, we don't > have pet names for them yet, like Unicode does, but their numeric > values are 0-255. Note that it in no way limits encodings, characters, > or scripts which can be used with MicroPython, because just like > Unicode, it support concept of "surrogate pairs" (but we don't call it > like that) - specifically, smaller code points may comprise bigger > groupings. But unlike Unicode, we don't stipulate format, value or > other constraints on how these "surrogate pairs"-alikes are formed, > leaving that to users. This is effectively what the Python 2 str type does, and it's a recipe for data driven latent defects. You inevitably end up concatenating strings using different code spaces, or else splitting strings between surrogate pairs rather than on the proper boundaries, etc. The abstraction presented to users by the str type *must* be the full range of Unicode code points as atomic units. Storing those internally as UTF-8 rather than as fixed width code points as CPython does is an experiment worth trying, since you don't have the same C level backwards compatibility constraints we do. But limiting the str type to a single code page per process is not an acceptable constraint in a Python 3 implementation. Regards, Nick. > > > -- > Best regards, > Paul mailto:pmis...@gmail.com > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [numpy wishlist] Interpreter support for temporary elision in third-party classes
Julian Taylor wrote: tp_can_elide receives two objects and returns one of three values: * can work inplace, operation is associative * can work inplace but not associative * cannot work inplace Does it really need to be that complicated? Isn't it sufficient just to ask the object potentially being overwritten whether it's okay to overwrite it? I.e. a parameterless method returning a boolean. -- Greg ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
Eli Bendersky wrote: > While we're at it, Clang in nearing a stage where it can compile C and C++ > on Windows *with ABI-compatibility to MSVC* (yes, even C++) -- see > href="http://clang.llvm.org/docs/MSVCCompatibility.html";>http://clang.llvm.org/docs/MSVCCompatibility.html > for more details. Could > this help? Possibly. "cl-clang" is exciting and I hope distutils will support it one day. Clang is not well known among Windows users as it is among users of "Unix" (Apple, Linux, FreeBSD, et al.) It would be even better if Python were bundled with Clang on Windows. The MinGW-based "SciPy toolchain" has ABI compatibility with MSVC only for C (and Fortran), not C++. Differences from vanilla MinGW is mainly static linkage of the MinGW runtime, different stack alignment (4 bytes instead of 16), and it links with msvcr91.dll instead of msvcrt.dll. Sturla ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
On Fri, Jun 6, 2014 at 4:32 PM, Sturla Molden wrote: > Brian Curtin wrote: > > > Well we're certainly not going to assume such a thing. I know people do > > that, but many don't (I never have). > > If Python 2.7 users are left with a dead compiler on Windows, they will > find a solution. For example, Enthought is already bundling their Python > distribution with gcc 2.8.1 on Windows. > While we're at it, Clang in nearing a stage where it can compile C and C++ on Windows *with ABI-compatibility to MSVC* (yes, even C++) -- see http://clang.llvm.org/docs/MSVCCompatibility.html for more details. Could this help? Eli ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [numpy wishlist] Interpreter support for temporary elision in third-party classes
On Fri, Jun 6, 2014 at 11:33 PM, Sturla Molden wrote: > Nathaniel Smith wrote: > >> The proposal in my initial email requires zero pthreads, and is >> substantially more effective. (Your proposal reduces only the alloc >> overhead for large arrays; mine reduces both alloc and memory access >> overhead for boyh large and small arrays.) > > My suggestion prevents the kernel from zeroing pages in the middle of a > computation, which is an important part. It would also be an optimiation > the Python interpreter could benefit from indepently of NumPy, by allowing > reuse of allocated memory pages within CPU bound portions of the Python > code. And no, the method I suggested does not only work for large arrays. Small allocations are already recycled within process and don't touch the kernel, so your method doesn't affect them at all. My guess is that PyMalloc is unlikely to start spawning background threads any time soon, but if you'd like to propose it maybe you should start a new thread for that? > If we really want to take out the memory access overhead, we need to > consider lazy evaluation. E.g. a context manager that collects a symbolic > expression and triggers evaluation on exit: > > with numpy.accelerate: > x = > y = > z = > # evaluation of x,y,z happens here Using an alternative evaluation engine is indeed another way to optimize execution, which is why projects like numexpr, numba, theano, etc. exist. But this is basically switching to a different language in a different VM. I think people will keep running plain-old-CPython code for some time yet, and the point of this thread is that there's some low-hanging fruit for making all *that* code faster. -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
On Fri, Jun 6, 2014 at 11:43 PM, Sturla Molden wrote: > Brett Cannon wrote: > >> Nope. A new minor release of Python is a massive undertaking which is why >> we have saved ourselves the hassle of doing a Python 2.8 or not giving a >> clear signal as to when Python 2.x will end as a language. > > Why not just define Python 2.8 as Python 2.7 except with a newer compiler? > I cannot see why that would be massive undertaking, if changing compiler > for 2.7 is neccesary anyway. This would require recompiling all packages on OS X and Linux, even though nothing had changed. -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
Brian Curtin wrote: > Well we're certainly not going to assume such a thing. I know people do > that, but many don't (I never have). If Python 2.7 users are left with a dead compiler on Windows, they will find a solution. For example, Enthought is already bundling their Python distribution with gcc 2.8.1 on Windows. Sturla ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
On Jun 6, 2014 6:01 PM, "Sturla Molden" wrote: > > Brian Curtin wrote: > > > Adding features into 3.x is already not enough of a carrot on the > > stick for many users. Intentionally leaving 2.7 on a dead compiler is > > like beating them with the stick. > > Those who want to build extensions on Windows will just use MinGW > (currently GCC 2.8.2) instead. Well we're certainly not going to assume such a thing. I know people do that, but many don't (I never have). ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
Martin v. Löwis wrote: > Am 06.06.14 22:13, schrieb Paul Moore: >> From >> http://www.visualstudio.com/en-us/downloads/visual-studio-14-ctp-vs >> >> """ >> Currently, Visual Studio "14" CTPs have known compatibility issues >> with previous releases of Visual Studio and should not be installed >> side-by-side on the same computer. >> """ > > I also found > > http://support.microsoft.com/kb/2967191 > > which is more specific about this issue: > > '''There are known issues when you install Visual Studio "14" CTP > 14.0.21730.1 DP on the same computer as Visual Studio 2013. While we expect > that > an uninstallation of Visual Studio "14" and then a repair of Visual Studio > 2013 > should fix these issues, our safest recommendation is to install Visual Studio > "14" in a VM, a VHD, a fresh computer, or another non-production test-only > computer that does not have Visual Studio 2013 on it. All of these Visual > Studio > side-by-side issues are expected to be fixed soon. Somebody ran a test to see how well the install/uninstall/repair scenario works, and it isn't that great. There are a lot of teams who contribute to Visual Studio, and not all of them have updated their installers yet (my team included...). Unfortunately, it all happened too close to the release to fix it for this version, hence the recommendation. Eventually, VS 14 will be safe to install side-by-side with earlier versions. Chances are it is safe enough with VS 2010 or VS 2012 - it's the one-version-prior that's causing the most trouble. > There is an installation block in this Visual Studio "14" CTP that will > prevent > installation on a computer where an earlier version of Visual Studio is > already > installed. To disable the block that will put the computer in an > un-recommended > state, add the value "BlockerOverride" to the registry: > HKLM\SOFTWARE\Microsoft\DevDiv\vs\Servicing''' > > So it seems to me that switching to VS 14 at this point in time is not > possible. > > Of course, Steve could certainly maintain a Mercurial clone in his > hg.python.org > sandbox that has all the necessary changes done, so people won't have to redo > the porting over and over. That's what I had in mind. [Earlier post] > 1. what is the availability of the compiler during the testing phase, >and what will it be immediately after the testing ends (where >traditionally people would have to buy licenses, or wait for VS >Express to be released)? It's freely available now as part of Visual Studio, and all the pre-release releases will include everything. The last release (RC or whatever they decide to call it this time) should have a go-live license, though it will also be time bombed. I believe Express will be released at the same time as the paid versions. > 2. what is the risk of installing a beta compiler on what might >otherwise be a "production" developer system? In particular, could >it interfere with other VS installations, and could it require a >complete system reinstall when the final release of VC 14 is >available? Answered above. It's as risky as it always is, though as I mentioned, VC 14 may well be fine against VC 10. Build-to-build upgrades may not be supported between pre-release versions, but typically RC to RTM upgrades are supported. > 3. what is the chance of the final release being delayed beyond >the planned release date of Python 3.5? Microsoft has a bad >track record of meeting release dates (or the tradition of not >announcing any for that reason); the blog says that it will >be available "sometime in 2015". Now, Python 3.5 might appear >November 2015, so what do we do if VS 2015 is not released >by the time 3.5b1 is planned? We keep the VS 2010 files around and make sure they keep working. This is the biggest risk of the whole plan, but I believe that there's enough of a gap between when VS 14 is planned to release (which I know, but can't share) and when Python 3.5 is planned (which I don't know, but have a semi-informed guess). Is Python 3.5b1 being built with VS 14 RC (hypothetically) a blocking issue? Do we need to resolve that now or can it wait until it happens? > Regards, > Martin Cheers, Steve ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
Brian Curtin wrote: > Adding features into 3.x is already not enough of a carrot on the > stick for many users. Intentionally leaving 2.7 on a dead compiler is > like beating them with the stick. Those who want to build extensions on Windows will just use MinGW (currently GCC 2.8.2) instead. NumPy and SciPy are planning a switch to a GCC based toolchain with static linkage of the MinGW runtime on Windows. It is carefully configured to be binary compatible with VS2008 on Python 2.7. The major reason for this is to use gfortran also on Windows. But the result will be a GCC based toolchain that anyone can use to build extensions on Windows. Sturla ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
Brett Cannon wrote: > Nope. A new minor release of Python is a massive undertaking which is why > we have saved ourselves the hassle of doing a Python 2.8 or not giving a > clear signal as to when Python 2.x will end as a language. Why not just define Python 2.8 as Python 2.7 except with a newer compiler? I cannot see why that would be massive undertaking, if changing compiler for 2.7 is neccesary anyway. Sturla ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [numpy wishlist] Interpreter support for temporary elision in third-party classes
Nathaniel Smith wrote: > The proposal in my initial email requires zero pthreads, and is > substantially more effective. (Your proposal reduces only the alloc > overhead for large arrays; mine reduces both alloc and memory access > overhead for boyh large and small arrays.) My suggestion prevents the kernel from zeroing pages in the middle of a computation, which is an important part. It would also be an optimiation the Python interpreter could benefit from indepently of NumPy, by allowing reuse of allocated memory pages within CPU bound portions of the Python code. And no, the method I suggested does not only work for large arrays. If we really want to take out the memory access overhead, we need to consider lazy evaluation. E.g. a context manager that collects a symbolic expression and triggers evaluation on exit: with numpy.accelerate: x = y = z = # evaluation of x,y,z happens here Sturla ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Internal representation of strings and Micropython
On 7 June 2014 00:52, Paul Sokolovsky wrote: > > At heart, this is exactly what the Python 3 "str" type is. The > > universal convention is "code points". > > Yes. Except for one small detail - Python3 specifies these code points > to be Unicode code points. And Unicode is a very bloated thing. > > But if we drop that "Unicode" stipulation, then it's also exactly what > MicroPython implements. Its "str" type consists of codepoints, we don't > have pet names for them yet, like Unicode does, but their numeric > values are 0-255. Note that it in no way limits encodings, characters, > or scripts which can be used with MicroPython, because just like > Unicode, it support concept of "surrogate pairs" (but we don't call it > like that) - specifically, smaller code points may comprise bigger > groupings. But unlike Unicode, we don't stipulate format, value or > other constraints on how these "surrogate pairs"-alikes are formed, > leaving that to users. I think you've missed my point. There is absolutely nothing conceptually bloaty about what a Python 3 string is. It's just like a 7-bit ASCII string, except each entry can be from a larger table. When you index into a Python 3 string, you get back exactly *one valid entry* from the Unicode code point table. That plus the length of the string, plus the guarantee of immutability gives everything needed to layer the rest of the string functionality on top. There are no surrogate pairs - each code point is standalone (unlike code *units*). It is conceptually very simple. The implementation may be difficult (if you're trying to do better than 4 bytes per code point) but the concept is dead simple. If the MicroPython string type requires people *using* it to deal with surrogates (i.e. indexing could return a value that is not a valid Unicode code point) then it will have broken the conceptual simplicity of the Python 3 string type (and most certainly can't be considered in any way compatible). Tim Delaney ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
On Fri, Jun 6, 2014 at 11:42 PM, wrote: > On Sat, Jun 07, 2014 at 05:33:45AM +1000, Chris Angelico wrote: > >> > Is it really any difference in maintenance if you just stop applying >> > updates to 2.7 and switch to 2.8? If 2.8 is really just 2.7 with a >> > new compiler then there should be no functional difference between >> > doing that and doing a 2.7.whatever except all of the tooling that >> > relies on the compiler not to change in micro releases won’t >> > suddenly break and freak out. > >> If the only difference between 2.7 and 2.8 is the compiler used on >> Windows, what happens on Linux and other platforms? A Python 2.8 would >> have to be materially different from Python 2.7, not just binarily >> incompatible on one platform. > > Grrmph, that's fair. Perhaps a final alternative is simply continuing > the 2.7 series with a stale compiler, as a kind of carrot on a stick to > encourage users to upgrade? Gating 2.7 life on the natural decline of > its supported compiler/related ecosystem seems somehow quite a gradual > and natural demise.. :) Adding features into 3.x is already not enough of a carrot on the stick for many users. Intentionally leaving 2.7 on a dead compiler is like beating them with the stick. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
Hi. On 6.6.2014. 21:46, Guido van Rossum wrote: A reminder: https://lh5.googleusercontent.com/-d4rF0qJPskQ/U0qpNjP5GoI/PW0/4RF_7zy3esY/w1118-h629-no/Python28.jpg *ROFL* Subtle, ain't he? *gdr* Best regards, Jurko Gospodnetić ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
Am 06.06.14 22:13, schrieb Paul Moore: > From http://www.visualstudio.com/en-us/downloads/visual-studio-14-ctp-vs > > """ > Currently, Visual Studio "14" CTPs have known compatibility issues > with previous releases of Visual Studio and should not be installed > side-by-side on the same computer. > """ I also found http://support.microsoft.com/kb/2967191 which is more specific about this issue: '''There are known issues when you install Visual Studio "14" CTP 14.0.21730.1 DP on the same computer as Visual Studio 2013. While we expect that an uninstallation of Visual Studio "14" and then a repair of Visual Studio 2013 should fix these issues, our safest recommendation is to install Visual Studio "14" in a VM, a VHD, a fresh computer, or another non-production test-only computer that does not have Visual Studio 2013 on it. All of these Visual Studio side-by-side issues are expected to be fixed soon. There is an installation block in this Visual Studio "14" CTP that will prevent installation on a computer where an earlier version of Visual Studio is already installed. To disable the block that will put the computer in an un-recommended state, add the value "BlockerOverride" to the registry: HKLM\SOFTWARE\Microsoft\DevDiv\vs\Servicing''' So it seems to me that switching to VS 14 at this point in time is not possible. Of course, Steve could certainly maintain a Mercurial clone in his hg.python.org sandbox that has all the necessary changes done, so people won't have to redo the porting over and over. Regards, Marti ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
On Sat, Jun 7, 2014 at 5:42 AM, wrote: > Perhaps a final alternative is simply continuing > the 2.7 series with a stale compiler, as a kind of carrot on a stick to > encourage users to upgrade? More likely, what would happen is that there'd be an alternate distribution of Python 2.7 (eg ActiveState), which would be language-compatible with python.org 2.7, but built with a different compiler, and therefore unable to use extensions built for python.org's 2.7. One way or another, pain will happen. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
On Sat, Jun 07, 2014 at 05:33:45AM +1000, Chris Angelico wrote: > > Is it really any difference in maintenance if you just stop applying > > updates to 2.7 and switch to 2.8? If 2.8 is really just 2.7 with a > > new compiler then there should be no functional difference between > > doing that and doing a 2.7.whatever except all of the tooling that > > relies on the compiler not to change in micro releases won’t > > suddenly break and freak out. > If the only difference between 2.7 and 2.8 is the compiler used on > Windows, what happens on Linux and other platforms? A Python 2.8 would > have to be materially different from Python 2.7, not just binarily > incompatible on one platform. Grrmph, that's fair. Perhaps a final alternative is simply continuing the 2.7 series with a stale compiler, as a kind of carrot on a stick to encourage users to upgrade? Gating 2.7 life on the natural decline of its supported compiler/related ecosystem seems somehow quite a gradual and natural demise.. :) David ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
Am 06.06.14 21:20, schrieb "Martin v. Löwis": > 2. what is the risk of installing a beta compiler on what might >otherwise be a "production" developer system? In particular, could >it interfere with other VS installations, and could it require a >complete system reinstall when the final release of VC 14 is >available? I found an official answer here: http://www.visualstudio.com/en-us/downloads/visual-studio-14-ctp-vs "Installing a CTP release will place a computer in an unsupported state. For that reason, we recommend only installing CTP releases in a virtual machine, or on a computer that is available for reformatting." So there is no promise that you will not need to reformat the system during the evolution of the compiler. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
On 6 June 2014 20:20, "Martin v. Löwis" wrote: > 2. what is the risk of installing a beta compiler on what might >otherwise be a "production" developer system? In particular, could >it interfere with other VS installations, and could it require a >complete system reinstall when the final release of VC 14 is >available? From http://www.visualstudio.com/en-us/downloads/visual-studio-14-ctp-vs """ Currently, Visual Studio "14" CTPs have known compatibility issues with previous releases of Visual Studio and should not be installed side-by-side on the same computer. """ It also states that installing the CTP on a PC puts that PC into "Unsupported" state. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
A reminder: https://lh5.googleusercontent.com/-d4rF0qJPskQ/U0qpNjP5GoI/PW0/4RF_7zy3esY/w1118-h629-no/Python28.jpg -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
On Sat, Jun 7, 2014 at 5:36 AM, Donald Stufft wrote: > Well it’d contain bug fixes and whatever other sorts of things you’d put > into a 2.7.whatever release. So they’d still want to upgrade to 2.8 since > that’ll have bug fixes. But it's not a potentially-breaking change. For example, on Debian Wheezy, there are a huge number of packages that depend on "python (<< 2.8)", because they expect Python 2.7 and *not* Python 2.8. A newer version 2.7 will satisfy that; a version 2.8 won't, because it's entirely possible that 2.8 will have something that's significantly different. That's what version numbers mean; Python follows the standard three-part convention, where you upgrade automatically only within the last part of the number. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
Am 06.06.14 20:25, schrieb Brian Curtin: > We're going to have to change it at some point, otherwise we're going > to have people in 2018 scrambling to find VS2008, which will be 35 > versions too old by then. Not sure whether you picked 2018 deliberately: extended support for VS2008 Professional ends on April 10, 2018. In any case, the extension problem will occur regardless of what you do: - if you switch compilers within 2.7, applications may crash - if you switch compilers and declare it 2.8, you extensions might not be available precompiled for some time (in particular if the developers of some package have abandoned 2.7) - if you don't switch compilers, availability of the tool chain will be terrible. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
On Jun 6, 2014, at 3:33 PM, Chris Angelico wrote: > On Sat, Jun 7, 2014 at 5:11 AM, Donald Stufft wrote: >> Is it really any difference in maintenance if you just stop applying updates >> to >> 2.7 and switch to 2.8? If 2.8 is really just 2.7 with a new compiler then >> there >> should be no functional difference between doing that and doing a >> 2.7.whatever >> except all of the tooling that relies on the compiler not to change in micro >> releases won’t suddenly break and freak out. > > If the only difference between 2.7 and 2.8 is the compiler used on > Windows, what happens on Linux and other platforms? A Python 2.8 would > have to be materially different from Python 2.7, not just binarily > incompatible on one platform. > > ChrisA > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/donald%40stufft.io Well it’d contain bug fixes and whatever other sorts of things you’d put into a 2.7.whatever release. So they’d still want to upgrade to 2.8 since that’ll have bug fixes. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
On Sat, Jun 7, 2014 at 5:11 AM, Donald Stufft wrote: > Is it really any difference in maintenance if you just stop applying updates > to > 2.7 and switch to 2.8? If 2.8 is really just 2.7 with a new compiler then > there > should be no functional difference between doing that and doing a 2.7.whatever > except all of the tooling that relies on the compiler not to change in micro > releases won’t suddenly break and freak out. If the only difference between 2.7 and 2.8 is the compiler used on Windows, what happens on Linux and other platforms? A Python 2.8 would have to be materially different from Python 2.7, not just binarily incompatible on one platform. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
Am 06.06.14 19:31, schrieb Brian Curtin: >> If that's a non-issue, or if we can actually drop XP support, I'm all for it. > > Extended support ended in April of this year, so I think we should put > XP as unsupported for 3.5 in PEP 11 - > http://legacy.python.org/dev/peps/pep-0011/ > > I seem to remember that we were waiting for this anyway. We don't actually need to explicitly put XP there, as PEP 11 ties our support to the Microsoft product life cycle. XP is not supported by Python anymore. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
Am 06.06.14 17:41, schrieb Steve Dower: > Hi all > > I would like to propose moving Python 3.5 to use Visual C++ 14.0 as > the main compiler. This is fine with me, but I'm worried about the precise timing of doing so. I assume that you would plan to do this moving before VC++ 14 is actually released. This worries me for three reasons: 1. what is the availability of the compiler during the testing phase, and what will it be immediately after the testing ends (where traditionally people would have to buy licenses, or wait for VS Express to be released)? 2. what is the risk of installing a beta compiler on what might otherwise be a "production" developer system? In particular, could it interfere with other VS installations, and could it require a complete system reinstall when the final release of VC 14 is available? 3. what is the chance of the final release being delayed beyond the planned release date of Python 3.5? Microsoft has a bad track record of meeting release dates (or the tradition of not announcing any for that reason); the blog says that it will be available "sometime in 2015". Now, Python 3.5 might appear November 2015, so what do we do if VS 2015 is not released by the time 3.5b1 is planned? Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
On Jun 6, 2014, at 3:09 PM, Brian Curtin wrote: > On Fri, Jun 6, 2014 at 11:08 PM, Donald Stufft wrote: >> >> On Jun 6, 2014, at 3:04 PM, Brian Curtin wrote: >> >>> On Fri, Jun 6, 2014 at 10:56 PM, wrote: On Fri, Jun 06, 2014 at 10:49:24PM +0400, Brian Curtin wrote: > None of the options are particularly good, but yes, I think that's an > option we have to consider. We're supporting 2.7.x for 6 more years on > a compiler that is already 6 years old. Surely that is infinitely less desirable than simply bumping the minor version? >>> >>> It's definitely not desirable, but "simply" bumping the minor version >>> is not A Thing. >> >> Why? I mean even if it’s the same thing as 2.7 just with an updated >> compiler that seems like a better answer than having to deal with >> 2.7.whatever suddenly breaking all C exts. > > Because then we have to maintain 2.8 at a time when no one even wants > to maintain 2.7? Is it really any difference in maintenance if you just stop applying updates to 2.7 and switch to 2.8? If 2.8 is really just 2.7 with a new compiler then there should be no functional difference between doing that and doing a 2.7.whatever except all of the tooling that relies on the compiler not to change in micro releases won’t suddenly break and freak out. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
On Fri, Jun 6, 2014 at 11:08 PM, Donald Stufft wrote: > > On Jun 6, 2014, at 3:04 PM, Brian Curtin wrote: > >> On Fri, Jun 6, 2014 at 10:56 PM, wrote: >>> On Fri, Jun 06, 2014 at 10:49:24PM +0400, Brian Curtin wrote: >>> None of the options are particularly good, but yes, I think that's an option we have to consider. We're supporting 2.7.x for 6 more years on a compiler that is already 6 years old. >>> >>> Surely that is infinitely less desirable than simply bumping the minor >>> version? >> >> It's definitely not desirable, but "simply" bumping the minor version >> is not A Thing. > > Why? I mean even if it’s the same thing as 2.7 just with an updated > compiler that seems like a better answer than having to deal with > 2.7.whatever suddenly breaking all C exts. Because then we have to maintain 2.8 at a time when no one even wants to maintain 2.7? ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
On Jun 6, 2014, at 3:04 PM, Brian Curtin wrote: > On Fri, Jun 6, 2014 at 10:56 PM, wrote: >> On Fri, Jun 06, 2014 at 10:49:24PM +0400, Brian Curtin wrote: >> >>> None of the options are particularly good, but yes, I think that's an >>> option we have to consider. We're supporting 2.7.x for 6 more years on >>> a compiler that is already 6 years old. >> >> Surely that is infinitely less desirable than simply bumping the minor >> version? > > It's definitely not desirable, but "simply" bumping the minor version > is not A Thing. Why? I mean even if it’s the same thing as 2.7 just with an updated compiler that seems like a better answer than having to deal with 2.7.whatever suddenly breaking all C exts. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
On Fri Jun 06 2014 at 2:59:24 PM, wrote: > On Fri, Jun 06, 2014 at 10:49:24PM +0400, Brian Curtin wrote: > > > None of the options are particularly good, but yes, I think that's an > > option we have to consider. We're supporting 2.7.x for 6 more years on > > a compiler that is already 6 years old. > > Surely that is infinitely less desirable than simply bumping the minor > version? > Nope. A new minor release of Python is a massive undertaking which is why we have saved ourselves the hassle of doing a Python 2.8 or not giving a clear signal as to when Python 2.x will end as a language. -Brett ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
On Fri, Jun 6, 2014 at 10:56 PM, wrote: > On Fri, Jun 06, 2014 at 10:49:24PM +0400, Brian Curtin wrote: > >> None of the options are particularly good, but yes, I think that's an >> option we have to consider. We're supporting 2.7.x for 6 more years on >> a compiler that is already 6 years old. > > Surely that is infinitely less desirable than simply bumping the minor > version? It's definitely not desirable, but "simply" bumping the minor version is not A Thing. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Division of tool labour in porting Python 2 code to 2/3
On Fri Jun 06 2014 at 2:29:13 PM, Terry Reedy wrote: > On 6/6/2014 12:37 PM, Brett Cannon wrote: > > After Glyph and Alex's email about their asks for assisting in writing > > Python 2/3 code, it got me thinking about where in the toolchain various > > warnings and such should go in order to help direct energy to help > > develop whatever future toolchain to assist in porting. > > > > There seems to be three places where issues are/can be caught once a > > project has embarked down the road of 2/3 source compatibility: > > > > 1. -3 warnings > > 2. Some linter tool > > 3. Failing tests > > > > -3 warnings are things that we know are flat-out wrong and do not cause > > massive compatibility issues in the stdlib. For instance, warning that > > buffer() is not in Python 3 is a py3k warning -- Glyph made a mistake > > when he asked for it as a new warning -- is a perfect example of > > something that isn't excessively noisy and won't cause issues when > > people run with it. > > > > But what about warning about classic classes? The stdlib is full of them > > and they were purposefully left alone for compatibility reasons. But > > there is a subtle semantic difference between classic and new-style > > classes, > > A non-subtle difference is that old style classes do not have .__new__. > I just ran into this when backporting an Idle test to 2.7. (I rewrote > the test to avoid diverging the code). In retrospect, perhaps we should > have added a global 'new-class future' -C switch, like -Q, and made sure > that stdlib worked either way. People running 2and3 code could then run > 2.x with the switch. Is is possible to add this now? > I consider changing the CLI out of bounds in a bugfix release as it's part of the API of Python. > > > and so 2/3 code should consider switching > > I do not not understand what you mean without a switch or furture > statement available to switch from old to new in 2.7. > Run a 2to3 fixer that changes all of their classes to new-style. > > > (this is when people > > chime in saying "this is why we want a 2.8 release!", but that still > > isn't happening). If this were made a py3k warning in 2.7 then the > > stdlib itself would spew out warnings which we can't change due to > > compatibility, so that makes it not useful > > (http://bugs.python.org/issue21231). > > Don't issue the warning if the class is in the stdlib. > If the warning is issued *after* creating class C: > > f = C.__module__.__file__ > if classic(C) and (not 'lib' in f or 'site_packages' in f): >warn(...) > > On Windows, the directory is 'Lib'; I presume it is lowercased > everywhere. If not, adjust. > That's just asking for trouble. I don't want to be import-dependent like that in the stdlib. > > > But as part of a lint tool specific > > to Python 2.7 that kind of warning would not be an issue and is easily > > managed and integrated into CI setups to make sure there are no > regressions. > > > > Lastly, there are things like string/unicode comparisons. > > http://bugs.python.org/issue21401 has a patch from VIctor which warns > > when comparing strings and unicode in Python 2.7. Much like the classic > > classes example, the stdlib becomes rather noisy due to APIs that handle > > either/or, etc. But unlike the classic classes example, you just can't > > systematically verify that two variables are always going to be str vs. > > unicode in Python 2.7 if they aren't literals. If people want to > > implement type constraint graphs for 2.7 code to help find them then > > that's great, but I personally don't have that kind of time. In this > > instance it would seem like relying on a project's unit tests to find > > this sort of problem is the best option. > > > > With those three levels in mind, where do we draw the line between these > > levels? Take for instance the print statement. Right now there is no > > warning with -3. Do we add one and then update the 2.7 stdlib to prevent > > warnings being generated by the stdlib? > > Make conditional as with class. > > We *could* change 'print s' to the exactly equivalent 'print(s)' > (perhaps half the cases); 'print r, s' to "print('%s %s' % (r,s)), > 'print '', y' to "print(' %s' % y), and so on. However, 'print > >>self.stdout, x', etc, does not translate to a pseudo-call. It would > need transltion to "self.stdout.write(x+'\n')". Grepping 2.7.6 lib/*.py > for print gives 1341 hits, with at least 1000 being actual print > statements. > Yep, which is why I don't want to do a 2to3 run on the stdlib to get rid of them. I also want to minimize conditional checks as it leads to potential issues of people thinking it's okay not to change things when there are in actually differences (e.g. I don't want to promote classic classes or native strings if it can be helped for the vast majority of users). -Brett > > > Or do we add it to some linter > > tool to pick up when people accidentally leave one in their code? > > > The reason I ask is since this
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
On Fri, Jun 06, 2014 at 10:49:24PM +0400, Brian Curtin wrote: > None of the options are particularly good, but yes, I think that's an > option we have to consider. We're supporting 2.7.x for 6 more years on > a compiler that is already 6 years old. Surely that is infinitely less desirable than simply bumping the minor version? David ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
On 06.06.2014 20:49, Brian Curtin wrote: > On Fri, Jun 6, 2014 at 10:41 PM, M.-A. Lemburg wrote: >> On 06.06.2014 20:25, Brian Curtin wrote: >>> On Fri, Jun 6, 2014 at 10:19 PM, Chris Angelico wrote: On Sat, Jun 7, 2014 at 4:12 AM, Steve Dower wrote: > Chris Angelico wrote: >> On Sat, Jun 7, 2014 at 1:41 AM, Steve Dower >> wrote: >>> What this means for Python is that C extensions for Python 3.5 and >>> later can be built using any version of MSVC from 14.0 and later. >> >> Oh, if only this had been available for 2.7!! Actually... this means >> that 14.0 would be a good target for a compiler change for 2.7.x, if >> such a change is ever acceptable. > > Maybe, but I doubt it will ever be acceptable :) Well, there were discussions. Since Python 2.7's support is far exceeding the Microsoft promise of support for the compiler it was built on, there's going to be a problem, one way or the other. I don't know how that's going to end up being resolved. >>> >>> We're going to have to change it at some point, otherwise we're going >>> to have people in 2018 scrambling to find VS2008, which will be 35 >>> versions too old by then. No matter what we do here, we're going to >>> have a tough PR situation, but we have to make something workable. I'd >>> rather cause a hassle than outright kill extensions. >>> >>> I would probably prefer we aim for VS 14 for 3.5, and then explore >>> making the same change for the 2.7.x release that comes after 3.5.0 >>> comes out. Lessons learned and all that. >> >> Are you sure that's an option ? Changing the compiler the stock >> Python from python.org is built with will most likely render >> existing Python extensions built for 2.7.x with x < (release that comes >> after 3.5.0) broken, so users and installation tools will end up >> having to pay close attention to the patch level version of Python >> they are using... which is something we wanted to avoid after >> we ran into this situation with 1.5.1 and 1.5.2 a few years ago. > > None of the options are particularly good, but yes, I think that's an > option we have to consider. We're supporting 2.7.x for 6 more years on > a compiler that is already 6 years old. Something less than awesome > for everyone involved is going to have to happen to make that > possible. Perhaps we could combine this with the breakage that a Python 2.7.10 would introduce due to the two digit patch level release version ;-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jun 06 2014) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2014-05-28: Released mxODBC.Connect 2.1.0 ... http://egenix.com/go56 2014-07-02: Python Meeting Duesseldorf ... 26 days to go : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
On Fri, Jun 6, 2014 at 10:41 PM, M.-A. Lemburg wrote: > On 06.06.2014 20:25, Brian Curtin wrote: >> On Fri, Jun 6, 2014 at 10:19 PM, Chris Angelico wrote: >>> On Sat, Jun 7, 2014 at 4:12 AM, Steve Dower >>> wrote: Chris Angelico wrote: > On Sat, Jun 7, 2014 at 1:41 AM, Steve Dower > wrote: >> What this means for Python is that C extensions for Python 3.5 and later >> can be built using any version of MSVC from 14.0 and later. > > Oh, if only this had been available for 2.7!! Actually... this means that > 14.0 would be a good target for a compiler change for 2.7.x, if such a > change is ever acceptable. Maybe, but I doubt it will ever be acceptable :) >>> >>> Well, there were discussions. Since Python 2.7's support is far >>> exceeding the Microsoft promise of support for the compiler it was >>> built on, there's going to be a problem, one way or the other. I don't >>> know how that's going to end up being resolved. >> >> We're going to have to change it at some point, otherwise we're going >> to have people in 2018 scrambling to find VS2008, which will be 35 >> versions too old by then. No matter what we do here, we're going to >> have a tough PR situation, but we have to make something workable. I'd >> rather cause a hassle than outright kill extensions. >> >> I would probably prefer we aim for VS 14 for 3.5, and then explore >> making the same change for the 2.7.x release that comes after 3.5.0 >> comes out. Lessons learned and all that. > > Are you sure that's an option ? Changing the compiler the stock > Python from python.org is built with will most likely render > existing Python extensions built for 2.7.x with x < (release that comes > after 3.5.0) broken, so users and installation tools will end up > having to pay close attention to the patch level version of Python > they are using... which is something we wanted to avoid after > we ran into this situation with 1.5.1 and 1.5.2 a few years ago. None of the options are particularly good, but yes, I think that's an option we have to consider. We're supporting 2.7.x for 6 more years on a compiler that is already 6 years old. Something less than awesome for everyone involved is going to have to happen to make that possible. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
On 06.06.2014 20:25, Brian Curtin wrote: > On Fri, Jun 6, 2014 at 10:19 PM, Chris Angelico wrote: >> On Sat, Jun 7, 2014 at 4:12 AM, Steve Dower >> wrote: >>> Chris Angelico wrote: On Sat, Jun 7, 2014 at 1:41 AM, Steve Dower wrote: > What this means for Python is that C extensions for Python 3.5 and later > can be built using any version of MSVC from 14.0 and later. Oh, if only this had been available for 2.7!! Actually... this means that 14.0 would be a good target for a compiler change for 2.7.x, if such a change is ever acceptable. >>> >>> Maybe, but I doubt it will ever be acceptable :) >> >> Well, there were discussions. Since Python 2.7's support is far >> exceeding the Microsoft promise of support for the compiler it was >> built on, there's going to be a problem, one way or the other. I don't >> know how that's going to end up being resolved. > > We're going to have to change it at some point, otherwise we're going > to have people in 2018 scrambling to find VS2008, which will be 35 > versions too old by then. No matter what we do here, we're going to > have a tough PR situation, but we have to make something workable. I'd > rather cause a hassle than outright kill extensions. > > I would probably prefer we aim for VS 14 for 3.5, and then explore > making the same change for the 2.7.x release that comes after 3.5.0 > comes out. Lessons learned and all that. Are you sure that's an option ? Changing the compiler the stock Python from python.org is built with will most likely render existing Python extensions built for 2.7.x with x < (release that comes after 3.5.0) broken, so users and installation tools will end up having to pay close attention to the patch level version of Python they are using... which is something we wanted to avoid after we ran into this situation with 1.5.1 and 1.5.2 a few years ago. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jun 06 2014) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2014-05-28: Released mxODBC.Connect 2.1.0 ... http://egenix.com/go56 2014-07-02: Python Meeting Duesseldorf ... 26 days to go : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Division of tool labour in porting Python 2 code to 2/3
On 6/6/2014 12:37 PM, Brett Cannon wrote: After Glyph and Alex's email about their asks for assisting in writing Python 2/3 code, it got me thinking about where in the toolchain various warnings and such should go in order to help direct energy to help develop whatever future toolchain to assist in porting. There seems to be three places where issues are/can be caught once a project has embarked down the road of 2/3 source compatibility: 1. -3 warnings 2. Some linter tool 3. Failing tests -3 warnings are things that we know are flat-out wrong and do not cause massive compatibility issues in the stdlib. For instance, warning that buffer() is not in Python 3 is a py3k warning -- Glyph made a mistake when he asked for it as a new warning -- is a perfect example of something that isn't excessively noisy and won't cause issues when people run with it. But what about warning about classic classes? The stdlib is full of them and they were purposefully left alone for compatibility reasons. But there is a subtle semantic difference between classic and new-style classes, A non-subtle difference is that old style classes do not have .__new__. I just ran into this when backporting an Idle test to 2.7. (I rewrote the test to avoid diverging the code). In retrospect, perhaps we should have added a global 'new-class future' -C switch, like -Q, and made sure that stdlib worked either way. People running 2and3 code could then run 2.x with the switch. Is is possible to add this now? > and so 2/3 code should consider switching I do not not understand what you mean without a switch or furture statement available to switch from old to new in 2.7. (this is when people chime in saying "this is why we want a 2.8 release!", but that still isn't happening). If this were made a py3k warning in 2.7 then the stdlib itself would spew out warnings which we can't change due to compatibility, so that makes it not useful (http://bugs.python.org/issue21231). Don't issue the warning if the class is in the stdlib. If the warning is issued *after* creating class C: f = C.__module__.__file__ if classic(C) and (not 'lib' in f or 'site_packages' in f): warn(...) On Windows, the directory is 'Lib'; I presume it is lowercased everywhere. If not, adjust. > But as part of a lint tool specific to Python 2.7 that kind of warning would not be an issue and is easily managed and integrated into CI setups to make sure there are no regressions. Lastly, there are things like string/unicode comparisons. http://bugs.python.org/issue21401 has a patch from VIctor which warns when comparing strings and unicode in Python 2.7. Much like the classic classes example, the stdlib becomes rather noisy due to APIs that handle either/or, etc. But unlike the classic classes example, you just can't systematically verify that two variables are always going to be str vs. unicode in Python 2.7 if they aren't literals. If people want to implement type constraint graphs for 2.7 code to help find them then that's great, but I personally don't have that kind of time. In this instance it would seem like relying on a project's unit tests to find this sort of problem is the best option. With those three levels in mind, where do we draw the line between these levels? Take for instance the print statement. Right now there is no warning with -3. Do we add one and then update the 2.7 stdlib to prevent warnings being generated by the stdlib? Make conditional as with class. We *could* change 'print s' to the exactly equivalent 'print(s)' (perhaps half the cases); 'print r, s' to "print('%s %s' % (r,s)), 'print '', y' to "print(' %s' % y), and so on. However, 'print >>self.stdout, x', etc, does not translate to a pseudo-call. It would need transltion to "self.stdout.write(x+'\n')". Grepping 2.7.6 lib/*.py for print gives 1341 hits, with at least 1000 being actual print statements. > Or do we add it to some linter > tool to pick up when people accidentally leave one in their code? The reason I ask is since this is clear I'm willing to spearhead the tooling work we talked about at the language summit to make sure there's a clear path for people wanting to port which is as easy as (reasonably) possible, but I don't want to start on it until I have a clear indication of what people are going to be okay with. -- Terry Jan Reedy ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
On Fri, Jun 6, 2014 at 10:19 PM, Chris Angelico wrote: > On Sat, Jun 7, 2014 at 4:12 AM, Steve Dower wrote: >> Chris Angelico wrote: >>> On Sat, Jun 7, 2014 at 1:41 AM, Steve Dower >>> wrote: What this means for Python is that C extensions for Python 3.5 and later can be built using any version of MSVC from 14.0 and later. >>> >>> Oh, if only this had been available for 2.7!! Actually... this means that >>> 14.0 would be a good target for a compiler change for 2.7.x, if such a >>> change is ever acceptable. >> >> Maybe, but I doubt it will ever be acceptable :) > > Well, there were discussions. Since Python 2.7's support is far > exceeding the Microsoft promise of support for the compiler it was > built on, there's going to be a problem, one way or the other. I don't > know how that's going to end up being resolved. We're going to have to change it at some point, otherwise we're going to have people in 2018 scrambling to find VS2008, which will be 35 versions too old by then. No matter what we do here, we're going to have a tough PR situation, but we have to make something workable. I'd rather cause a hassle than outright kill extensions. I would probably prefer we aim for VS 14 for 3.5, and then explore making the same change for the 2.7.x release that comes after 3.5.0 comes out. Lessons learned and all that. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
On Sat, Jun 7, 2014 at 4:12 AM, Steve Dower wrote: > Chris Angelico wrote: >> On Sat, Jun 7, 2014 at 1:41 AM, Steve Dower >> wrote: >>> What this means for Python is that C extensions for Python 3.5 and later >>> can be built using any version of MSVC from 14.0 and later. >> >> Oh, if only this had been available for 2.7!! Actually... this means that >> 14.0 would be a good target for a compiler change for 2.7.x, if such a >> change is ever acceptable. > > Maybe, but I doubt it will ever be acceptable :) Well, there were discussions. Since Python 2.7's support is far exceeding the Microsoft promise of support for the compiler it was built on, there's going to be a problem, one way or the other. I don't know how that's going to end up being resolved. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
Chris Angelico wrote: > On Sat, Jun 7, 2014 at 1:41 AM, Steve Dower wrote: >> What this means for Python is that C extensions for Python 3.5 and later can >> be built using any version of MSVC from 14.0 and later. > > Oh, if only this had been available for 2.7!! Actually... this means that > 14.0 would be a good target for a compiler change for 2.7.x, if such a change > is ever acceptable. Maybe, but I doubt it will ever be acceptable :) > To what extent is this compatibility going to be maintained? Is there a > guarantee that there'll be X versions (or X years) of cross-compilation > support? There are a few breaking changes in this version that are designed to standardize on a function-call based ABI, which should effectively be a life-long guarantee. The only promise I can make is this: when cross-compilation support is eventually broken, it will be due to something that nobody has been able to predict up until now. (Hopefully that's better than promising that it will be broken in the very next release.) > ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
Stefan Krah wrote: >Stefan Krah wrote: >> > * Will VS 14 be golden prior to Python 3.5's release? It would suck to >> > rely on a beta compiler.. :) >> >> This is my only concern, too. Otherwise, +1 for the switch. > >One more thing: Will the SDK 64-bit tools be available for the Express >Versions? They should be. If they're not, I'll certainly be making a noise about it (unless there's another, easier way to get the tools by then...) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
dw+python-...@hmmz.org wrote: > Speaking as a third party who aims to provide binary distributions for recent > Python releases on Windows, every new compiler introduces a licensing and > configuration headache. So I guess the questions are: > > * Does the ABI stability address some historical real world problem with > Python binary builds? (I guess possibly) Yes. It's very hard to explain to users that even though they've gone out and paid for Visual Studio 2013 Ultimate, they don't really have a C compiler that works with Python. This stability will eventually get us to a place where it doesn't matter what version of the compiler you have, though for a while people will obviously need the latest. (Another thing I'm working on is making sure that it's really easy to get the latest... lots of pieces to this puzzle.) > * Is the existing solution of third parties building under e.g. Mingw as > an option of last resort causing real world issues? It seems to work > for a lot of people, although I personally avoid it. I think it actually tends to solve more issues than it causes :( I want to fix that by making MSVC better for Python, rather than switching away to another toolset. > * Have other compiler vendors indicated they will change their ABI > environment to match VS under this new stability guarantee? If not, > then as yet there is no real world benefit here. I have no idea, but I hope they do (eventually they almost certainly will). I've already mentioned to our team that they should reach out to the other projects and try to help them move it along, though I have no idea if they have the time or contacts to manage that. FWIW, the stability guarantee was only announced this week, so there's a good chance that the gcc/clang/etc. teams aren't even aware of it yet. > * Has Python ever hit a showstopper release issue as a result of a bug > in MSVC? (I guess probably not). Not to my knowledge, and I'm certainly hoping to avoid it by keeping the builds coming regularly. I can't do an official buildbot for it (and probably can't even reuse the infrastructure) since I'm going to work against the latest internal version as much as I can and we get new builds almost daily. More likely, building Python will reveal showstopper issues that actually get fixed (and it has done in the past, though that was never publicised :) ) > * Will VS 14 be golden prior to Python 3.5's release? It would suck to > rely on a beta compiler.. :) I sure hope so. The current planning looks like it will (I'm assuming that Python 3.5 is going to be late next year, but I couldn't find a good reference). If things slip here, I'm going to be surrounded by very stressed people, which is not much fun. So I hope it'll be done! At worst, VS 14 RC (or whatever label it gets) will probably be released under a "go live" licence. If anything is dramatically broken at that point, we'll know and it should be fixed, or we know that it's going to be around for a while regardless and we can make the decision to either stick with VC10 or work around the issues. > Sorry for dunking water on this, but I've recently spent a ton of time > getting a > Microsoft build environment running, and it seems possible a new compiler may > not yet justify more effort if there is little tangible benefit. Not at all. I've spent far more time than I wanted to getting a build environment running for producing the Python 2.7 installers, and I spent just as long getting an environment for default going too. I'm personally a big fan of automating things like this, so you can also expect scripts (probably PowerShell) that will configure as much as possible. Cheers, Steve > David ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
On Fri, Jun 6, 2014 at 8:22 PM, Zachary Ware wrote: > On Fri, Jun 6, 2014 at 10:41 AM, Steve Dower > wrote: >> Thoughts/comments/concerns? > > My only concern is support for elderly versions of Windows, in > particular: XP. I seem to recall the last "let's update our MSVC > version" discussion dying off because of XP support. Even though MS > has abandoned it, I'm not sure whether we can yet. > > If that's a non-issue, or if we can actually drop XP support, I'm all for it. Extended support ended in April of this year, so I think we should put XP as unsupported for 3.5 in PEP 11 - http://legacy.python.org/dev/peps/pep-0011/ I seem to remember that we were waiting for this anyway. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [numpy wishlist] Interpreter support for temporary elision in third-party classes
On 06.06.2014 04:26, Greg Ewing wrote: > Nathaniel Smith wrote: > >> I'd be a little nervous about whether anyone has implemented, say, an >> iadd with side effects such that you can tell whether a copy was made, >> even if the object being copied is immediately destroyed. > > I can think of at least one plausible scenario where > this could occur: the operand is a view object that > wraps another object, and its __iadd__ method updates > that other object. > > In fact, now that I think about it, exactly this > kind of thing happens in numpy when you slice an > array! > > So the opt-in indicator would need to be dynamic, on > a per-object basis, rather than a type flag. > yes an opt-in indicator would need to receive both operand objects so it would need to be a slot in the object or number type object. Would the addition of a tp_can_elide slot to the object types be acceptable for this rather specialized case? tp_can_elide receives two objects and returns one of three values: * can work inplace, operation is associative * can work inplace but not associative * cannot work inplace Implementation could e.g. look about like this: TARGET(BINARY_SUBTRACT) { fl = left->obj_type->tp_can_elide fr = right->obj_type->tp_can_elide elide = 0 if (unlikely(fl)) { elide = fl(left, right) } else if (unlikely(fr)) { elide = fr(left, right) } if (unlikely(elide == YES) && left->refcnt == 1) { PyNumber_InPlaceSubtract(left, right) } else if (unlikely(elide == SWAPPABLE) && right->refcnt == 1) { PyNumber_InPlaceSubtract(right, left) } else { PyNumber_Subtract(left, right) } } ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
Stefan Krah wrote: > > * Will VS 14 be golden prior to Python 3.5's release? It would suck to > > rely on a beta compiler.. :) > > This is my only concern, too. Otherwise, +1 for the switch. One more thing: Will the SDK 64-bit tools be available for the Express Versions? Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [numpy wishlist] Interpreter support for temporary elision in third-party classes
On 6 Jun 2014 17:07, "Sturla Molden" wrote: > We would in total need two mutexes, one condition variable, a pthread, and > a heap. The proposal in my initial email requires zero pthreads, and is substantially more effective. (Your proposal reduces only the alloc overhead for large arrays; mine reduces both alloc and memory access overhead for boyh large and small arrays.) -n ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
On Fri, 06 Jun 2014 16:37:01 -, dw+python-...@hmmz.org wrote: > On Fri, Jun 06, 2014 at 03:41:22PM +, Steve Dower wrote: > > > [snip] > > Speaking as a third party who aims to provide binary distributions for > recent Python releases on Windows, every new compiler introduces a > licensing and configuration headache. So I guess the questions are: > > * Does the ABI stability address some historical real world problem with > Python binary builds? (I guess possibly) > > * Is the existing solution of third parties building under e.g. Mingw as > an option of last resort causing real world issues? It seems to work > for a lot of people, although I personally avoid it. > > * Have other compiler vendors indicated they will change their ABI > environment to match VS under this new stability guarantee? If not, > then as yet there is no real world benefit here. > > * Has Python ever hit a showstopper release issue as a result of a bug > in MSVC? (I guess probably not). > > * Will VS 14 be golden prior to Python 3.5's release? It would suck to > rely on a beta compiler.. :) > > > Sorry for dunking water on this, but I've recently spent a ton of time > getting a Microsoft build environment running, and it seems possible a > new compiler may not yet justify more effort if there is little tangible > benefit. If I understand correctly (but I may not as I'm not a windows dev) we're going to want to switch VS versions for 3.5 anyway, so switching to the cutting edge one but where Steve can be and is willing to be in a tight feedback loop with the developers sounds like a win to me. --David ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
dw+python-...@hmmz.org wrote: > * Has Python ever hit a showstopper release issue as a result of a bug > in MSVC? (I guess probably not). Yes, a PGO issue: http://bugs.python.org/issue15993 To be fair, in that issue I did not look if there's some undefined behavior in longobject.c. > * Will VS 14 be golden prior to Python 3.5's release? It would suck to > rely on a beta compiler.. :) This is my only concern, too. Otherwise, +1 for the switch. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Division of tool labour in porting Python 2 code to 2/3
After Glyph and Alex's email about their asks for assisting in writing Python 2/3 code, it got me thinking about where in the toolchain various warnings and such should go in order to help direct energy to help develop whatever future toolchain to assist in porting. There seems to be three places where issues are/can be caught once a project has embarked down the road of 2/3 source compatibility: 1. -3 warnings 2. Some linter tool 3. Failing tests -3 warnings are things that we know are flat-out wrong and do not cause massive compatibility issues in the stdlib. For instance, warning that buffer() is not in Python 3 is a py3k warning -- Glyph made a mistake when he asked for it as a new warning -- is a perfect example of something that isn't excessively noisy and won't cause issues when people run with it. But what about warning about classic classes? The stdlib is full of them and they were purposefully left alone for compatibility reasons. But there is a subtle semantic difference between classic and new-style classes, and so 2/3 code should consider switching (this is when people chime in saying "this is why we want a 2.8 release!", but that still isn't happening). If this were made a py3k warning in 2.7 then the stdlib itself would spew out warnings which we can't change due to compatibility, so that makes it not useful (http://bugs.python.org/issue21231). But as part of a lint tool specific to Python 2.7 that kind of warning would not be an issue and is easily managed and integrated into CI setups to make sure there are no regressions. Lastly, there are things like string/unicode comparisons. http://bugs.python.org/issue21401 has a patch from VIctor which warns when comparing strings and unicode in Python 2.7. Much like the classic classes example, the stdlib becomes rather noisy due to APIs that handle either/or, etc. But unlike the classic classes example, you just can't systematically verify that two variables are always going to be str vs. unicode in Python 2.7 if they aren't literals. If people want to implement type constraint graphs for 2.7 code to help find them then that's great, but I personally don't have that kind of time. In this instance it would seem like relying on a project's unit tests to find this sort of problem is the best option. With those three levels in mind, where do we draw the line between these levels? Take for instance the print statement. Right now there is no warning with -3. Do we add one and then update the 2.7 stdlib to prevent warnings being generated by the stdlib? Or do we add it to some linter tool to pick up when people accidentally leave one in their code? The reason I ask is since this is clear I'm willing to spearhead the tooling work we talked about at the language summit to make sure there's a clear path for people wanting to port which is as easy as (reasonably) possible, but I don't want to start on it until I have a clear indication of what people are going to be okay with. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
On Fri, Jun 06, 2014 at 03:41:22PM +, Steve Dower wrote: > [snip] Speaking as a third party who aims to provide binary distributions for recent Python releases on Windows, every new compiler introduces a licensing and configuration headache. So I guess the questions are: * Does the ABI stability address some historical real world problem with Python binary builds? (I guess possibly) * Is the existing solution of third parties building under e.g. Mingw as an option of last resort causing real world issues? It seems to work for a lot of people, although I personally avoid it. * Have other compiler vendors indicated they will change their ABI environment to match VS under this new stability guarantee? If not, then as yet there is no real world benefit here. * Has Python ever hit a showstopper release issue as a result of a bug in MSVC? (I guess probably not). * Will VS 14 be golden prior to Python 3.5's release? It would suck to rely on a beta compiler.. :) Sorry for dunking water on this, but I've recently spent a ton of time getting a Microsoft build environment running, and it seems possible a new compiler may not yet justify more effort if there is little tangible benefit. David ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Internal representation of strings and Micropython
Hello, On Fri, 06 Jun 2014 11:59:31 -0400 Terry Reedy wrote: [] > The other problem is that a small slice view of a large object keeps > the large object alive, so a view user needs to think carefully about > whether to make a copy or create a view, and later to copy views to > delete the base object. This is not for beginners. Yes, so it doesn't make sense to add such feature to any of existing APIs. However, as I pointed in another mail, it would make lot of sense to add iterator-based string API (because if dict methods were *switched* to iterators, why can't string have them *as alternative*), and for their return values, it would be ~ natural to return "string views", especially if it's clearly and explicitly described that if user wants to store them, they should be explicitly copied via str(view). One reason against this would be of course API bloat. But API bloat happens all the time, for example compare this modest proposal http://bugs.python.org/issue21180 with what's going to be actually implemented: http://legacy.python.org/dev/peps/pep-0467/#alternate-constructors . -- Best regards, Paul mailto:pmis...@gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
On Fri, Jun 6, 2014 at 10:41 AM, Steve Dower wrote: > Thoughts/comments/concerns? My only concern is support for elderly versions of Windows, in particular: XP. I seem to recall the last "let's update our MSVC version" discussion dying off because of XP support. Even though MS has abandoned it, I'm not sure whether we can yet. If that's a non-issue, or if we can actually drop XP support, I'm all for it. -- Zach ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
On 6 June 2014 16:41, Steve Dower wrote: > Basically, what I am offering to do is: > > * Update the files in PCBuild to work with Visual Studio "14" > * Make any code changes necessary to build with VC14 > * Regularly test the latest Python source with the latest MSVC builds and > report issues/suggestions to the MSVC team > * Keep all changes in a separate (public) repo until early next year when > we're getting close to the final VS "14" release > > What I am asking anyone else to do is: > > * Nothing +1 from me. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Internal representation of strings and Micropython
On 06/06/2014 05:59 PM, Terry Reedy wrote: The other problem is that a small slice view of a large object keeps the large object alive, so a view user needs to think carefully about whether to make a copy or create a view, and later to copy views to delete the base object. This is not for beginners. And this was important enough that Java 7 actually removed the long-standing feature of String.substring creating a string that shares the character array with the original. http://java-performance.info/changes-to-string-java-1-7-0_06/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] asyncio/Tulip: use CPython as the new upstream
On Jun 06, 2014, at 04:47 PM, MRAB wrote: >Isn't this a little like when bool, True and False were added to >Python 2.2.1, a bugfix release, an act that is, I believe, now regarded >as a mistake not to be repeated? Yes, that was a mistake, but the case under discussion is different. With True/False, it was a runtime-wide change that affected every Python program, and there was no such "special dispensation". -Barry ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] asyncio/Tulip: use CPython as the new upstream
On Fri, Jun 6, 2014 at 8:47 AM, MRAB wrote: > On 2014-06-06 10:31, Victor Stinner wrote: > >> Hi, >> >> I added a new BaseEventLoop.is_closed() method to Tulip and Python >> 3.5 to fix an issue (see Tulip issue 169 for the detail). The problem >> is that I don't want to add this method to Python 3.4 because usually >> we don't add new methods in minor versions of Python (future version >> 3.4.2 in this case). >> >> Guido just wrote in the issue: "Actually for asyncio we have special >> dispensation to push new features to minor releases (until 3.5). >> Please push to 3.4 so the source code is the same everywhere (except >> selectors.py, which is not covered by the exception)." >> >> I disagree with Guido. I would prefer to start to maintain a >> different branch for Python 3.4, because I consider that only >> bugfixes should be applied to Python 3.4. >> >> [snip] > > Isn't this a little like when bool, True and False were added to > Python 2.2.1, a bugfix release, an act that is, I believe, now regarded > as a mistake not to be repeated? > It's a little like that, but it's also a little unlike that -- asyncio is explicitly accepted in the stdlib with "provisional" status which allows changes like this. Regarding the workflow, I'd really like asyncio to be able to move faster than the rest of the stdlib, at least until 3.5 is fixed. Working in the Tulip repo is much easier for me than working in the CPython repo, so I'd like to keep the workflow of Tulip -> 3.4 -> 3.5 as long as possible. I also specifically consider selectors.py subject to a *different* workflow -- for that module the workflow should be 3.5 -> Tulip. If Tulip's update_stdlib.sh script's prompts to copy this file are too distracting, I can hack the script to be silent about this file if it detects that the CPython repo is 3.4. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Summary of Python tracker Issues
ACTIVITY SUMMARY (2014-05-30 - 2014-06-06) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open4650 (+15) closed 28802 (+52) total 33452 (+67) Open issues with patches: 2127 Issues opened (48) == #21614: Case sensitivity problem in multiprocessing. http://bugs.python.org/issue21614 opened by ColinPDavidson #21615: Curses bug report for Python 2.7 and Python 3.2 http://bugs.python.org/issue21615 opened by eclectic9509 #21616: argparse explodes with nargs='*' and a tuple metavar http://bugs.python.org/issue21616 opened by vvas #21617: importlib reload can fail with AttributeError if module remove http://bugs.python.org/issue21617 opened by ned.deily #21619: Cleaning up a subprocess with a broken pipe http://bugs.python.org/issue21619 opened by vadmium #21621: Add note to 3.x What's New re Idle changes in bugfix releases http://bugs.python.org/issue21621 opened by terry.reedy #21622: ctypes.util incorrectly fails for libraries without DT_SONAME http://bugs.python.org/issue21622 opened by Jeremy.Huntwork #21623: build ssl failed use vs2010 express http://bugs.python.org/issue21623 opened by Mo.Jia #21624: Idle: polish htests http://bugs.python.org/issue21624 opened by terry.reedy #21625: help()'s more-mode is frustrating http://bugs.python.org/issue21625 opened by nedbat #21626: Add options width and compact to pickle cli http://bugs.python.org/issue21626 opened by barcc #21627: Concurrently closing files and iterating over the open files d http://bugs.python.org/issue21627 opened by sstewartgallus #21629: clinic.py --converters fails http://bugs.python.org/issue21629 opened by serhiy.storchaka #21632: Idle: sychronize text files across versions as appropriate. http://bugs.python.org/issue21632 opened by terry.reedy #21633: Argparse does not propagate HelpFormatter class to subparsers http://bugs.python.org/issue21633 opened by Michael.Cohen #21635: difflib.SequenceMatcher stores matching blocks as tuples, not http://bugs.python.org/issue21635 opened by drevicko #21642: "_ if 1else _" does not compile http://bugs.python.org/issue21642 opened by Joshua.Landau #21644: Optimize bytearray(int) constructor to use calloc() http://bugs.python.org/issue21644 opened by haypo #21645: test_read_all_from_pipe_reader() of test_asyncio hangs on Free http://bugs.python.org/issue21645 opened by haypo #21646: Add tests for turtle.ScrolledCanvas http://bugs.python.org/issue21646 opened by ingrid #21647: Idle unittests: make gui, mock switching easier. http://bugs.python.org/issue21647 opened by terry.reedy #21648: urllib urlopener leaves open sockets for FTP connection http://bugs.python.org/issue21648 opened by Claudiu.Popa #21649: Mention "Recommendations for Secure Use of TLS and DTLS" http://bugs.python.org/issue21649 opened by pitrou #21650: add json.tool option to avoid alphabetic sort of fields http://bugs.python.org/issue21650 opened by Pavel.Kazlou #21652: Python 2.7.7 regression in mimetypes module on Windows http://bugs.python.org/issue21652 opened by foom #21655: Write Unit Test for Vec2 class in the Turtle Module http://bugs.python.org/issue21655 opened by Lita.Cho #21656: Create test coverage for TurtleScreenBase in Turtle http://bugs.python.org/issue21656 opened by Lita.Cho #21657: pip.get_installed_distributions() Does not return packages in http://bugs.python.org/issue21657 opened by Adam.Matan #21658: __m128, can't build 3.4.1 with intel 14.0.0 http://bugs.python.org/issue21658 opened by aom #21659: IDLE: One corner calltip case http://bugs.python.org/issue21659 opened by serhiy.storchaka #21660: Substitute @TOKENS@ from sysconfig variables, for python-confi http://bugs.python.org/issue21660 opened by haubi #21664: multiprocessing leaks temporary directories pymp-xxx http://bugs.python.org/issue21664 opened by yjhong #21665: 2.7.7 ttk widgets not themed http://bugs.python.org/issue21665 opened by les.bothwell #21666: Argparse exceptions should include which argument has a proble http://bugs.python.org/issue21666 opened by v+python #21667: Clarify status of O(1) indexing semantics of str objects http://bugs.python.org/issue21667 opened by ncoghlan #21668: The select and time modules uses libm functions without linkin http://bugs.python.org/issue21668 opened by fornwall #21669: Custom error messages when print & exec are used as statements http://bugs.python.org/issue21669 opened by ncoghlan #21670: Add repr to shelve.Shelf http://bugs.python.org/issue21670 opened by Claudiu.Popa #21671: CVE-2014-0224: OpenSSL upgrade to 1.0.1h on Windows required http://bugs.python.org/issue21671 opened by lambacck #21672: Python for Windows 2.7.7: Path Configuration File No Longer Wo http://bugs.python.org/issue21672 opened by jblairpdx #21673: Idle: hilite search terms in hits in Find in
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
On Jun 6, 2014, at 11:41 AM, Steve Dower wrote: > words +1 from me. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [numpy wishlist] Interpreter support for temporary elision in third-party classes
Julian Taylor wrote: > The problem with this approach is that it is already difficult enough to > handle memory in numpy. I would not do this in a way that complicates memory management in NumPy. I would just replace malloc and free with temporarily cached versions. From the perspective of NumPy the API should be the same. > Having a cache that potentially stores gigabytes > of memory out of the users sight will just make things worse. Buffer don't need to stay in cache forver, just long enough to allow resue within an expression. We are probably talking about delaying the call to free with just a few microseconds. We could e.g. have a setup like this: NumPy thread on "malloc": - tries to grab memory off the internal heap - calls system malloc on failure NumPy thread on "free": - returns a buffer to the internal heap - signals a condition Background daemonic GC thread: - wakes after sleeping on the condition - sleeps for another N microseconds (N = magic number) - flushes or shrinks the internal heap with system free - goes back to sleeping on the condition It can be implemented with the same API as malloc and free, and plugged directly into the existing NumPy code. We would in total need two mutexes, one condition variable, a pthread, and a heap. Sturla ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
On Sat, Jun 7, 2014 at 1:41 AM, Steve Dower wrote: > What this means for Python is that C extensions for Python 3.5 and later can > be built using any version of MSVC from 14.0 and later. Oh, if only this had been available for 2.7!! Actually... this means that 14.0 would be a good target for a compiler change for 2.7.x, if such a change is ever acceptable. To what extent is this compatibility going to be maintained? Is there a guarantee that there'll be X versions (or X years) of cross-compilation support? ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Internal representation of strings and Micropython
On 6/6/2014 4:53 AM, Hrvoje Niksic wrote: On 06/04/2014 05:52 PM, Mark Lawrence wrote: Out of idle curiosity is there anything that stops MicroPython, or any other implementation for that matter, from providing views of a string rather than copying every time? IIRC memoryviews in CPython rely on the buffer protocol at the C API level, so since strings don't support this protocol you can't take a memoryview of them. Could this actually be implemented in the future, is the underlying C code just too complicated, or what? Memory view of Unicode strings is controversial for two reasons: 1. It exposes the internal representation of the string. If memoryviews of strings were supported in Python 3, PEP 393 would not have been possible (without breaking that feature). 2. Even if it were OK to expose the internal representation, it might not be what the users expect. For example, memoryview("Hrvoje") would return a view of a 6-byte buffer, while memoryview("Nikšić") would return a view of a 12-byte UCS-2 buffer. The user of a memory view might expect to get UCS-2 (or UCS-4, or even UTF-8) in all cases. An implementation that decided to export strings as memory views might be forced to make a decision about internal representation of strings, and then stick to it. The byte objects don't have these issues, which is why in Python 2.7 memoryview("foo") works just fine, as does memoryview(b"foo") in Python 3. The other problem is that a small slice view of a large object keeps the large object alive, so a view user needs to think carefully about whether to make a copy or create a view, and later to copy views to delete the base object. This is not for beginners. -- Terry Jan Reedy ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Moving Python 3.5 on Windows to a new compiler
Hi all I would like to propose moving Python 3.5 to use Visual C++ 14.0 as the main compiler. The first CTP of Visual Studio "14" was released earlier this week: http://blogs.msdn.com/b/vcblog/archive/2014/06/03/visual-studio-14-ctp.aspx The major feature of interest in this version of MSVC is a new policy to maintain binary compatibility for the CRT into the future. (There will be a blog about this soon, but I didn't want to hold up getting the discussion started here.) What this means for Python is that C extensions for Python 3.5 and later can be built using any version of MSVC from 14.0 and later. Those who are aware of the current state of affairs where you need to use a matching compiler will hopefully see how big an improvement this will be. It is also likely that other compilers will have an easier time providing compatibility with this new CRT, making it simpler and more reliable to build extensions with LLVM or GCC against an MSVC CPython. The other major benefit is that both products are at points in their development where changes can be made. Being a Microsoft employee, I have the ability to test Python builds regularly against the daily MSVC builds and to file bugs directly to the VC team (crashes, incorrect code generation, incorrect linking, performance regressions, etc.). This is a great opportunity to make sure that our needs are covered by the compiler team - it's also a good chance to raise any particular missing features that would be beneficial. My internal testing shows that the core code is almost fully compatible and builds successfully with only trivial modifications (some CRT variables are now macros with a leading underscore). The project files need updating, but I am willing to do this as part of any migration. There may also be some work required for external dependencies, since I did not test these, but I am also willing to do that. Basically, what I am offering to do is: * Update the files in PCBuild to work with Visual Studio "14" * Make any code changes necessary to build with VC14 * Regularly test the latest Python source with the latest MSVC builds and report issues/suggestions to the MSVC team * Keep all changes in a separate (public) repo until early next year when we're getting close to the final VS "14" release What I am asking anyone else to do is: * Nothing Thoughts/comments/concerns? Cheers, Steve ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] asyncio/Tulip: use CPython as the new upstream
On 2014-06-06 10:31, Victor Stinner wrote: Hi, I added a new BaseEventLoop.is_closed() method to Tulip and Python 3.5 to fix an issue (see Tulip issue 169 for the detail). The problem is that I don't want to add this method to Python 3.4 because usually we don't add new methods in minor versions of Python (future version 3.4.2 in this case). Guido just wrote in the issue: "Actually for asyncio we have special dispensation to push new features to minor releases (until 3.5). Please push to 3.4 so the source code is the same everywhere (except selectors.py, which is not covered by the exception)." I disagree with Guido. I would prefer to start to maintain a different branch for Python 3.4, because I consider that only bugfixes should be applied to Python 3.4. [snip] Isn't this a little like when bool, True and False were added to Python 2.2.1, a bugfix release, an act that is, I believe, now regarded as a mistake not to be repeated? ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Internal representation of strings and Micropython
On Fri, Jun 6, 2014 at 8:15 PM, Paul Sokolovsky wrote: > I'm sorry if I was somehow related to that, my > bringing in the formal language spec was more a rhetorical figure, a > response to people claiming O(1) requirement. This was exactly why this whole discussion came up, though. We were debating on the uPy bug tracker about how important O(1) indexing is; I then came to python-list to try to get some solid data from which to debate; and then the discussion jumped here to python-dev for more solid explanations. The spec wasn't perfectly clear, and now it's being made clearer: O(N) indexing does not violate Python's spec, ergo uPy is allowed to use UTF-8 as its internal representation, as long as script-visible behaviour is correct. It'll be interesting to see when it's done (I'm currently working on that implementation, bit by bit) and to run the CPython benchmarks on it. It's been a fruitful and interesting discussion, and the formal language spec is key to it. No need to apologize! ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Internal representation of strings and Micropython
Hello, On Fri, 6 Jun 2014 21:48:41 +1000 Tim Delaney wrote: > On 6 June 2014 21:34, Paul Sokolovsky wrote: > > > > > On Fri, 06 Jun 2014 20:11:27 +0900 > > "Stephen J. Turnbull" wrote: > > > > > Paul Sokolovsky writes: > > > > > > > That kinda means "string is atomic", instead of your > > > > "characters are atomic". > > > > > > I would be very surprised if a language that behaved that way was > > > called a "Python subset". No indexing, no slicing, no regexps, no > > > .split(), no .startswith(), no sorted() or .sort(), ...!? > > > > > > If that's not what you mean by "string is atomic", I think you're > > > using very confusing terminology. > > > > I'm sorry if I didn't mention it, or didn't make it clear enough - > > it's all about layering. > > > > On level 0, you treat strings verbatim, and can write some subset of > > apps (my point is that even this level allows to write lot enough > > apps). Let's call this set A0. > > > > On level 1, you accept that there's some universal enough > > conventions for some chars, like space or newline. And you can > > write set of apps A1 > A0. > > > > At heart, this is exactly what the Python 3 "str" type is. The > universal convention is "code points". Yes. Except for one small detail - Python3 specifies these code points to be Unicode code points. And Unicode is a very bloated thing. But if we drop that "Unicode" stipulation, then it's also exactly what MicroPython implements. Its "str" type consists of codepoints, we don't have pet names for them yet, like Unicode does, but their numeric values are 0-255. Note that it in no way limits encodings, characters, or scripts which can be used with MicroPython, because just like Unicode, it support concept of "surrogate pairs" (but we don't call it like that) - specifically, smaller code points may comprise bigger groupings. But unlike Unicode, we don't stipulate format, value or other constraints on how these "surrogate pairs"-alikes are formed, leaving that to users. -- Best regards, Paul mailto:pmis...@gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] asyncio/Tulip: use CPython as the new upstream
On Fri, 06 Jun 2014 10:05:52 -0400, Antoine Pitrou wrote: > Le 06/06/2014 07:00, R. David Murray a écrit : > > > > I don't have any opinion on the workflow. > > > > My understanding is that part of the purpose of the "provisional" > > designation is to allow faster evolution (read: fixing) of an API before > > the library becomes non-provisional. Thus I agree with Guido here, and > > will be doing something similar with at least one of the minor provisional > > email API features in 3.4.2 (unless I miss the cutoff again ... :( > > I would personally distinguish API fixes (compatibility-breaking > changes) from feature additions (new APIs). It doesn't look like the PEP directly addresses API changes in maintenance releases, and I suppose that should be fixed. I specifically want to fix this API before someone depends on it working the wrong way, which they would have to if I left it alone for the whole of the 3.4 series. (Issue 21091 for the curious.) --David ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] asyncio/Tulip: use CPython as the new upstream
Le 06/06/2014 07:00, R. David Murray a écrit : I don't have any opinion on the workflow. My understanding is that part of the purpose of the "provisional" designation is to allow faster evolution (read: fixing) of an API before the library becomes non-provisional. Thus I agree with Guido here, and will be doing something similar with at least one of the minor provisional email API features in 3.4.2 (unless I miss the cutoff again ... :( I would personally distinguish API fixes (compatibility-breaking changes) from feature additions (new APIs). Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Internal representation of strings and Micropython
On 06/06/2014 09:53, Hrvoje Niksic wrote: On 06/04/2014 05:52 PM, Mark Lawrence wrote: On 04/06/2014 16:32, Steve Dower wrote: If copying into a separate list is a problem (memory-wise), re.finditer('\\S+', string) also provides the same behaviour and gives me the sliced string, so there's no need to index for anything. Out of idle curiosity is there anything that stops MicroPython, or any other implementation for that matter, from providing views of a string rather than copying every time? IIRC memoryviews in CPython rely on the buffer protocol at the C API level, so since strings don't support this protocol you can't take a memoryview of them. Could this actually be implemented in the future, is the underlying C code just too complicated, or what? Memory view of Unicode strings is controversial for two reasons: 1. It exposes the internal representation of the string. If memoryviews of strings were supported in Python 3, PEP 393 would not have been possible (without breaking that feature). 2. Even if it were OK to expose the internal representation, it might not be what the users expect. For example, memoryview("Hrvoje") would return a view of a 6-byte buffer, while memoryview("Nikšić") would return a view of a 12-byte UCS-2 buffer. The user of a memory view might expect to get UCS-2 (or UCS-4, or even UTF-8) in all cases. An implementation that decided to export strings as memory views might be forced to make a decision about internal representation of strings, and then stick to it. The byte objects don't have these issues, which is why in Python 2.7 memoryview("foo") works just fine, as does memoryview(b"foo") in Python 3. Thanks for the explanation :) -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence --- This email is free from viruses and malware because avast! Antivirus protection is active. http://www.avast.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Internal representation of strings and Micropython
Hello, On Fri, 06 Jun 2014 09:32:25 +0100 Mark Lawrence wrote: > On 04/06/2014 16:52, Mark Lawrence wrote: > > On 04/06/2014 16:32, Steve Dower wrote: > >> > >> If copying into a separate list is a problem (memory-wise), > >> re.finditer('\\S+', string) also provides the same behaviour and > >> gives me the sliced string, so there's no need to index for > >> anything. > >> > > > > Out of idle curiosity is there anything that stops MicroPython, or > > any other implementation for that matter, from providing views of a > > string rather than copying every time? IIRC memoryviews in CPython > > rely on the buffer protocol at the C API level, so since strings > > don't support this protocol you can't take a memoryview of them. > > Could this actually be implemented in the future, is the underlying > > C code just too complicated, or what? > > > > Anybody? I'd like to address this, and other, buffer manipulation optimization ideas I have for MicroPython at some time later. But as you suggest, it would possible to transparently have "strings-by-reference". The reasons MicroPython doesn't have such so far (and why I'm, as a uPy contributor, not ready to discuss them) is because they're optimization, and everyone knows what premature optimization is. [] -- Best regards, Paul mailto:pmis...@gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Internal representation of strings and Micropython
On 6 June 2014 21:34, Paul Sokolovsky wrote: > > On Fri, 06 Jun 2014 20:11:27 +0900 > "Stephen J. Turnbull" wrote: > > > Paul Sokolovsky writes: > > > > > That kinda means "string is atomic", instead of your "characters > > > are atomic". > > > > I would be very surprised if a language that behaved that way was > > called a "Python subset". No indexing, no slicing, no regexps, no > > .split(), no .startswith(), no sorted() or .sort(), ...!? > > > > If that's not what you mean by "string is atomic", I think you're > > using very confusing terminology. > > I'm sorry if I didn't mention it, or didn't make it clear enough - it's > all about layering. > > On level 0, you treat strings verbatim, and can write some subset of > apps (my point is that even this level allows to write lot enough > apps). Let's call this set A0. > > On level 1, you accept that there's some universal enough conventions > for some chars, like space or newline. And you can write set of > apps A1 > A0. > At heart, this is exactly what the Python 3 "str" type is. The universal convention is "code points". It's got nothing to do with encodings, or bytes. A Python string is simply a finite sequence of atomic code points - it is indexable, and it has a length. Once you have that, everything is layered on top of it. How the code points themselves are implemented is opaque and irrelevant other than the memory and performance consequences of the implementation decisions (for example, a string could be indexable by iterating from the start until you find the nth code point). Similarly the "bytes" type is a sequence of 8-bit bytes. Encodings are simply a way to transport code points via a byte-oriented transport. Tim Delaney ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Internal representation of strings and Micropython
On 6 June 2014 21:15, Paul Sokolovsky wrote: > Hello, > > On Thu, 5 Jun 2014 23:15:54 +1000 > Nick Coghlan wrote: > >> On 5 June 2014 22:37, Paul Sokolovsky wrote: >> > On Thu, 5 Jun 2014 22:20:04 +1000 >> > Nick Coghlan wrote: >> >> problems caused by trusting the locale encoding to be correct, but >> >> the startup code will need non-trivial changes for that to happen >> >> - the C.UTF-8 locale may even become widespread before we get >> >> there). >> > >> > ... And until those golden times come, it would be nice if Python >> > did not force its perfect world model, which unfortunately is not >> > based on surrounding reality, and let users solve their encoding >> > problems themselves - when they need, because again, one can go >> > quite a long way without dealing with encodings at all. Whereas now >> > Python3 forces users to deal with encoding almost universally, but >> > forcing a particular for all strings (which is again, doesn't >> > correspond to the state of surrounding reality). I already hear >> > response that it's good that users taught to deal with encoding, >> > that will make them write correct programs, but that's a bit far >> > away from the original aim of making it write "correct" programs >> > easy and pleasant. (And definition of "correct" vary.) >> >> As I've said before in other contexts, find me Windows, Mac OS X and >> JVM developers, or educators and scientists that are as concerned by >> the text model changes as folks that are primarily focused on Linux >> system (including network) programming, and I'll be more willing to >> concede the point. > > Well, but this question reduces to finding out (or specifying) who are > target audiences of Python. It always has been (with a bow to Guido) > forpost of scientific users (and probably even if there was mass exodus > of other categories of users will remain prominent in that role). But > Python has always had its share as system scripting language among > Perl-haters, and with Perl going flatline, I guess it's fair to say > that Python is major system scripting and service implementation > language. Correct - and the efforts of a number of core developers are focused on getting the Linux distros and major projects like OpenStack migrated. If other Linux users say "I'm not switching to Python 3 until after my distro has switched their own Python applications over", that's a perfectly reasonable course of action for them to take. After all, that approach to the adoption of new Python versions is a large part of why Python 2.6 is still so widely supported by library and framework developers: enterprise Linux distros haven't even finished migrating to Python 2.7 yet, let alone Python 3. (The other reason is that the language moratorium that was applied to Python 2.7 and 3.2 means that supporting back to Python 2.6 isn't that much harder than supporting 2.7 at this point in time). That said, the feedback from the early adopters of Python 3 on Linux is proving invaluable, and Linux users in general will benefit from their work as the distros move their infrastructure applications over. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Internal representation of strings and Micropython
Hello, On Fri, 06 Jun 2014 20:11:27 +0900 "Stephen J. Turnbull" wrote: > Paul Sokolovsky writes: > > > That kinda means "string is atomic", instead of your "characters > > are atomic". > > I would be very surprised if a language that behaved that way was > called a "Python subset". No indexing, no slicing, no regexps, no > .split(), no .startswith(), no sorted() or .sort(), ...!? > > If that's not what you mean by "string is atomic", I think you're > using very confusing terminology. I'm sorry if I didn't mention it, or didn't make it clear enough - it's all about layering. On level 0, you treat strings verbatim, and can write some subset of apps (my point is that even this level allows to write lot enough apps). Let's call this set A0. On level 1, you accept that there's some universal enough conventions for some chars, like space or newline. And you can write set of apps A1 > A0. On level 2, you add len(), and - oh magic - you now can center a string within fixed-size field, something you probably to as often as once a month, so hopefully that will keep you busy for few. On level 3, it indeed starts to smell Unicode, we get isdigit(), isalpha(), which require long boring tables, which hopefully can be compressed enough to fit in your pocket. On level 4, it's pumping up, with tolower() and friends, tables for which you carry around in suitcase. On level 5, everything is Unicode, what a bliss! You can even start pretending that no other levels exist (God created Unicode on a second day). On level 6, there're mind-boggling, ugly manual-use utilities to deal with internals of "magic" "working on its own for everyone" encoding to deal with stuff like code-point vs charecters vs surrogate pair vs grapheme separation, etc. So, once again, for me and some other people, it's not that bright idea to shoot for level 5 if levels 0-4 exist and well-proven pragmatic model. And level 6 is still there anyway. -- Best regards, Paul mailto:pmis...@gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Internal representation of strings and Micropython
Hello, On Thu, 5 Jun 2014 23:15:54 +1000 Nick Coghlan wrote: > On 5 June 2014 22:37, Paul Sokolovsky wrote: > > On Thu, 5 Jun 2014 22:20:04 +1000 > > Nick Coghlan wrote: > >> problems caused by trusting the locale encoding to be correct, but > >> the startup code will need non-trivial changes for that to happen > >> - the C.UTF-8 locale may even become widespread before we get > >> there). > > > > ... And until those golden times come, it would be nice if Python > > did not force its perfect world model, which unfortunately is not > > based on surrounding reality, and let users solve their encoding > > problems themselves - when they need, because again, one can go > > quite a long way without dealing with encodings at all. Whereas now > > Python3 forces users to deal with encoding almost universally, but > > forcing a particular for all strings (which is again, doesn't > > correspond to the state of surrounding reality). I already hear > > response that it's good that users taught to deal with encoding, > > that will make them write correct programs, but that's a bit far > > away from the original aim of making it write "correct" programs > > easy and pleasant. (And definition of "correct" vary.) > > As I've said before in other contexts, find me Windows, Mac OS X and > JVM developers, or educators and scientists that are as concerned by > the text model changes as folks that are primarily focused on Linux > system (including network) programming, and I'll be more willing to > concede the point. Well, but this question reduces to finding out (or specifying) who are target audiences of Python. It always has been (with a bow to Guido) forpost of scientific users (and probably even if there was mass exodus of other categories of users will remain prominent in that role). But Python has always had its share as system scripting language among Perl-haters, and with Perl going flatline, I guess it's fair to say that Python is major system scripting and service implementation language. To whom all features like memoryview, array.array, in-place input operations, etc. cater? To scientists? I'm sure most of them are just happy with stuffing "@jit" for their kernel functions. And scientist who bother with memoryviews for their data structures are system-level-ish programmers too. So, no wonder that Linux crowd cries at Python3 - it makes doing simple things unnecessarily complicated. > Windows, Mac OS X, and the JVM are all opinionated about the text > encodings to be used at platform boundaries (using UTF-16, UTF-8 and > UTF-16, respectively). By contrast, Linux (or, more accurately, POSIX) > says "well, it's configurable, but we won't provide a reliable > mechanism for finding out what the encoding is. So either guess as [] Yes, I understand complexity of developing cross-platform language with advanced features. By I may offer another look at all this activity: Python3 was brave enough to do revolution in its own world (catching a lot of its users by surprise), but surely not brave enough to do revolution around itself, by saying something like "We choose ONE, the most right, and even the most used (per bytes transferred) encoding as our standard I/O encoding. Grow up or explicitly specify encoding which you personally need.". Surely, it didn't to that - it makes no sense to fight the world. But then Python3 is sympathetic about Java's desire to use "UTF-16" instead of "right" encoding, and no so about Unix desire to treat encodings as a separate level from content (and treating Unicode by nothing else as yet another arbitrary encoding, which it is formally, and will be for a long time de-facto, however sad it is). So, maybe "cross-platform" should have mean "don't do implicit conversions". Because see, Python2 had a problem with implicit encoding conversion when str and unicode objects were mixed, and Python3 has problem with implicit conversions whenever str is used at all. Anyway, I appreciate detailed responses, and understand what you (Python3 developers) are trying to achieve, and appreciate your work, and hope it all work out. Each user has own concerns about Unicode. Mine are efficiency and layering. But once MicroPython has UTF-8 support I will be much more relaxed about it. Layering is harder to accept, but hopefully can be tackled too both on own mind's and technical sides. I hope other users will find their peace with Unicode too! [] -- Best regards, Paul mailto:pmis...@gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Internal representation of strings and Micropython
Paul Sokolovsky writes: > That kinda means "string is atomic", instead of your "characters are > atomic". I would be very surprised if a language that behaved that way was called a "Python subset". No indexing, no slicing, no regexps, no .split(), no .startswith(), no sorted() or .sort(), ...!? If that's not what you mean by "string is atomic", I think you're using very confusing terminology. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] asyncio/Tulip: use CPython as the new upstream
On 6 June 2014 19:31, Victor Stinner wrote: > Guido just wrote in the issue: "Actually for asyncio we have special > dispensation to push new features to minor releases (until 3.5). > Please push to 3.4 so the source code is the same everywhere (except > selectors.py, which is not covered by the exception)." > > I disagree with Guido. I would prefer to start to maintain a different > branch for Python 3.4, because I consider that only bugfixes should be > applied to Python 3.4. This is why PEP 411 was thrashed out: to let us split the dates of "make broadly available in the standard library" and "get ultra conservative with API changes". asyncio was added as a provisional module, so it can still get new features in 3.4.x maintenance releases - that's a far more minor change than the backwards compatibility breaks permitted by the PEP. The difference with selectors is that it was *not* added as a provisional module - it's subject to all the normal stability requirements. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] asyncio/Tulip: use CPython as the new upstream
On Fri, 06 Jun 2014 11:31:23 +0200, Victor Stinner wrote: > Hi, > > I added a new BaseEventLoop.is_closed() method to Tulip and Python 3.5 > to fix an issue (see Tulip issue 169 for the detail). The problem is > that I don't want to add this method to Python 3.4 because usually we > don't add new methods in minor versions of Python (future version > 3.4.2 in this case). > > Guido just wrote in the issue: "Actually for asyncio we have special > dispensation to push new features to minor releases (until 3.5). > Please push to 3.4 so the source code is the same everywhere (except > selectors.py, which is not covered by the exception)." > > I disagree with Guido. I would prefer to start to maintain a different > branch for Python 3.4, because I consider that only bugfixes should be > applied to Python 3.4. > > It's not the first change that cannot be applied on Python 3.4 (only > in Tulip and Python 3.5): the selectors module now also supports > devpoll on Solaris. It's annoying because the Tulip script > "update_stdlib.sh" used to synchronize Tulip and Python wants to > replace Lib/selectors.py in Python 3.4. I have to revert the change each time. > > I propose a new workflow: use Python default (future version 3.5) as > the new asyncio "upstream". Bugfixes would be applied as other Python > bugfixes: first in Python 3.4, than in Python 3.5. The > "update_stdlib.sh" script of Tulip should be modified to copy files > from Python default to Tulip (opposite of the current direction). > > Workflow: > > New feature: Python 3.5 => Tulip => Trollius > Bugfix: Python 3.4 => Python 3.5 => Tulip => Trollius > > I don't think that Tulip should have minor release just for bugfixes, > it would be a pain to maintain. Tulip is a third party module, it > doesn't have the same constraints than Python stdlib. > > What do you think? I don't have any opinion on the workflow. My understanding is that part of the purpose of the "provisional" designation is to allow faster evolution (read: fixing) of an API before the library becomes non-provisional. Thus I agree with Guido here, and will be doing something similar with at least one of the minor provisional email API features in 3.4.2 (unless I miss the cutoff again ... :( --David ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Internal representation of strings and Micropython
Steven D'Aprano wrote: I don't know about car engine controllers, but presumably they have diagnostic ports, and they may sometimes output text. If they output text, then at least hypothetically car mechanics in Russia might prefer their car to output "правда" and "ложный" rather than "true" and "false". From a bit of googling, it seems that engine controller diagnostic ports typically speak some kind of binary protocol. So it would be up to the software running on whatever was plugged into the port to display the information in the user's native language. E.g. this document lists a big pile of hex byte values and little or no text that I can see: https://law.resource.org/pub/us/cfr/ibr/005/sae.j1979.2002.pdf -- Greg ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Internal representation of strings and Micropython
Hello, On Thu, 5 Jun 2014 22:38:13 +1000 Nick Coghlan wrote: > On 5 June 2014 22:10, Stefan Krah wrote: > > Paul Sokolovsky wrote: > >> In this regard, I'm glad to participate in mind-resetting > >> discussion. So, let's reiterate - there's nothing like "the best", > >> "the only right", "the only correct", "righter than", "more > >> correct than" in CPython's implementation of Unicode storage. It > >> is *arbitrary*. Well, sure, it's not arbitrary, but based on > >> requirements, and these requirements match CPython's (implied) > >> usage model well enough. But among all possible sets of > >> requirements, CPython's requirements are no more valid that other > >> possible. And other set of requirement fairly clearly lead to > >> situation where CPython implementation is rejected as not correct > >> for those requirements at all. > > > > Several core-devs have said that using UTF-8 for MicroPython is > > perfectly okay. I also think it's the right choice and I hope that > > you guys come up with a very efficient implementation. > > Based on this discussion , I've also posted a draft patch aimed at > clarifying the relevant aspects of the data model section of the > language reference (http://bugs.python.org/issue21667). Thanks, it's very much appreciated. Though, the discussion there opened another can of worms. I'm sorry if I was somehow related to that, my bringing in the formal language spec was more a rhetorical figure, a response to people claiming O(1) requirement. So, it either should be in spec, or spec should be treated as such - something not specified means it's underspecified and implementation-dependent. I'm glad that the last point now explicitly pronounced by BDFL in the last comment of that ticket (http://bugs.python.org/issue21667#msg219824) > > Cheers, > Nick. > > -- > Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/pmiscml%40gmail.com -- Best regards, Paul mailto:pmis...@gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] asyncio/Tulip: use CPython as the new upstream
Hi, I added a new BaseEventLoop.is_closed() method to Tulip and Python 3.5 to fix an issue (see Tulip issue 169 for the detail). The problem is that I don't want to add this method to Python 3.4 because usually we don't add new methods in minor versions of Python (future version 3.4.2 in this case). Guido just wrote in the issue: "Actually for asyncio we have special dispensation to push new features to minor releases (until 3.5). Please push to 3.4 so the source code is the same everywhere (except selectors.py, which is not covered by the exception)." I disagree with Guido. I would prefer to start to maintain a different branch for Python 3.4, because I consider that only bugfixes should be applied to Python 3.4. It's not the first change that cannot be applied on Python 3.4 (only in Tulip and Python 3.5): the selectors module now also supports devpoll on Solaris. It's annoying because the Tulip script "update_stdlib.sh" used to synchronize Tulip and Python wants to replace Lib/selectors.py in Python 3.4. I have to revert the change each time. I propose a new workflow: use Python default (future version 3.5) as the new asyncio "upstream". Bugfixes would be applied as other Python bugfixes: first in Python 3.4, than in Python 3.5. The "update_stdlib.sh" script of Tulip should be modified to copy files from Python default to Tulip (opposite of the current direction). Workflow: New feature: Python 3.5 => Tulip => Trollius Bugfix: Python 3.4 => Python 3.5 => Tulip => Trollius I don't think that Tulip should have minor release just for bugfixes, it would be a pain to maintain. Tulip is a third party module, it doesn't have the same constraints than Python stdlib. What do you think? Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Internal representation of strings and Micropython
Hello, On Thu, 5 Jun 2014 22:21:30 +1000 Tim Delaney wrote: > On 5 June 2014 22:01, Paul Sokolovsky wrote: > > > > > All these changes are what let me dream on and speculate on > > possibility that Python4 could offer an encoding-neutral string type > > (which means based on bytes) > > > > To me, an "encoding neutral string type" means roughly "characters are > atomic", and the best representation we have for a "character" is a And for me it means exactly what "encoding neutral string type" moniker promises - that you should not make any assumption about its encoding. That kinda means "string is atomic", instead of your "characters are atomic". That's the most basic level, and you can write a big enough set of applications using it - for example, get some information from user, store in database, then show back to user at later time. [] > > Cheers, > > Tim Delaney -- Best regards, Paul mailto:pmis...@gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Internal representation of strings and Micropython
On 06/04/2014 05:52 PM, Mark Lawrence wrote: On 04/06/2014 16:32, Steve Dower wrote: If copying into a separate list is a problem (memory-wise), re.finditer('\\S+', string) also provides the same behaviour and gives me the sliced string, so there's no need to index for anything. Out of idle curiosity is there anything that stops MicroPython, or any other implementation for that matter, from providing views of a string rather than copying every time? IIRC memoryviews in CPython rely on the buffer protocol at the C API level, so since strings don't support this protocol you can't take a memoryview of them. Could this actually be implemented in the future, is the underlying C code just too complicated, or what? Memory view of Unicode strings is controversial for two reasons: 1. It exposes the internal representation of the string. If memoryviews of strings were supported in Python 3, PEP 393 would not have been possible (without breaking that feature). 2. Even if it were OK to expose the internal representation, it might not be what the users expect. For example, memoryview("Hrvoje") would return a view of a 6-byte buffer, while memoryview("Nikšić") would return a view of a 12-byte UCS-2 buffer. The user of a memory view might expect to get UCS-2 (or UCS-4, or even UTF-8) in all cases. An implementation that decided to export strings as memory views might be forced to make a decision about internal representation of strings, and then stick to it. The byte objects don't have these issues, which is why in Python 2.7 memoryview("foo") works just fine, as does memoryview(b"foo") in Python 3. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [numpy wishlist] Interpreter support for temporary elision in third-party classes
On 06.06.2014 04:18, Sturla Molden wrote: > On 05/06/14 22:51, Nathaniel Smith wrote: > >> This gets evaluated as: >> >> tmp1 = a + b >> tmp2 = tmp1 + c >> result = tmp2 / c >> >> All these temporaries are very expensive. Suppose that a, b, c are >> arrays with N bytes each, and N is large. For simple arithmetic like >> this, then costs are dominated by memory access. Allocating an N byte >> array requires the kernel to clear the memory, which incurs N bytes of >> memory traffic. > > It seems to be the case that a large portion of the run-time in Python > code using NumPy can be spent in the kernel zeroing pages (which the > kernel does for security reasons). > > I think this can also be seen as a 'malloc problem'. It comes about > because each new NumPy array starts with a fresh buffer allocated by > malloc. Perhaps buffers can be reused? > > Sturla > > Caching memory inside of numpy would indeed solve this issue too. There has even been a paper written on this which contains some more serious benchmarks than the laplace case which runs on very old hardware (and the inplace and out of place cases are actually not the same, one computes array/scalar the other array * (1 / scalar)): hiperfit.dk/pdf/Doubling.pdf "The result is an improvement of as much as 2.29 times speedup, on average 1.32 times speedup across a benchmark suite of 15 applications" The problem with this approach is that it is already difficult enough to handle memory in numpy. Having a cache that potentially stores gigabytes of memory out of the users sight will just make things worse. This would not be needed if we can come up with a way on how python can help out numpy in eliding the temporaries. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Internal representation of strings and Micropython
On 04/06/2014 16:52, Mark Lawrence wrote: On 04/06/2014 16:32, Steve Dower wrote: If copying into a separate list is a problem (memory-wise), re.finditer('\\S+', string) also provides the same behaviour and gives me the sliced string, so there's no need to index for anything. Out of idle curiosity is there anything that stops MicroPython, or any other implementation for that matter, from providing views of a string rather than copying every time? IIRC memoryviews in CPython rely on the buffer protocol at the C API level, so since strings don't support this protocol you can't take a memoryview of them. Could this actually be implemented in the future, is the underlying C code just too complicated, or what? Anybody? -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence --- This email is free from viruses and malware because avast! Antivirus protection is active. http://www.avast.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com