Re: [Python-Dev] Status of C compilers for Python on Windows
Stephen J. Turnbull: Python is open source. Nobody is objecting to somebody else doing this.[1] The problem here is simply that some somebody elses are trying to throw future work over the wall into python-dev space. If that's how it's seen at this point, then it sounds like the logical course of action for CPython-with-MinGW is to demonstrate feasibility in a fork. Get what currently works as a set of 80-something patches and PKGBUILD instructions into a single repository that is usable by a wider variety of people, in more distributions, etc. Set up as much CI as possible so every patch gets tested in as many configurations as we can. Experiment with extension compatibility and find out what is actually achievable, with or without needing to modify MinGW-w64 in the process. There are people, though evidently not much of python-dev, who have a need and desire to make this happen. It seems python-dev would rather have us spend zero time proposing changes that allow CPython itself to be built differently than today, and all of our time on MinGW extensions. I personally find 3 of the 4 combinations of how one could build CPython and how one could build extensions interesting and worth looking into for different reasons (the one that's uninteresting to me is the currently used all-MSVC method, due to its many limitations I listed earlier). Ray for example may only care about using MinGW for everything. Python's a big community with lots of room for different people to work on different aspects of the same set of problems. For the combination of MSVC Python and MinGW extensions that most of you have recommended we focus on, it would be more productive to engage with setuptools, distutils-sig, and likely numpy as well, instead of python-dev. My experience lies more in getting troublesome C codebases to build with MinGW than it does with the messy intricacies and backwards-compatibility requirements of Python extensions and package management however, so my ability to contribute on that side of things is more limited. I'll just note that every project I've ever had a patch for which improved functionality with a new compiler (whether GCC, MSVC, clang, icc or ifort, etc.) reacted with review and thanks for the patches, not why do you want to do this? pushback. If potential contributors have a desire to get it working in the first place, then they will also be invested in helping keep it working on an ongoing basis. Sincerely, Tony ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of C compilers for Python on Windows
Tony Kelman writes: If potential contributors have a desire to get it working in the first place, then they will also be invested in helping keep it working on an ongoing basis. Sure -- as long as it works for them, though, such potential contributors don't necessarily care if it works for anybody else. My experience (in other projects) is that allowing that level of commitment to be sufficient for inclusion in the maintained code base frequently results in bug reports from third parties who try to use the new feature and discover it *doesn't* work for them. The core maintainers then have a very unpleasant choice: to say we don't support that usage, or to deal with the problem on a continuing basis (because the same issues tend to regress repeatedly as the said invested contributors continue to modify their code on the same works for us basis). If that's how it's seen at this point, then it sounds like the logical course of action for CPython-with-MinGW is to demonstrate feasibility in a fork. Get what currently works as a set of 80-something patches and PKGBUILD instructions into a single repository that is usable by a wider variety of people, in more distributions, etc. Set up as much CI as possible so every patch gets tested in as many configurations as we can. Experiment with extension compatibility and find out what is actually achievable, with or without needing to modify MinGW-w64 in the process. Sounds good to me. You seem to think that's excessive, though: There are people, though evidently not much of python-dev, who have a need and desire to make this happen. Well, for starters, most of python-dev would rather avoid any contact whatsoever with Windows. I think part of the problem is that Windows developers *of* Python are *very* rare (fingers of one hand rare). Sure, there are many Windows-based Python developers, and quite of few of them do a fair amount to contribute to Python on Windows -- but they do that work in *Python*, not at a level where they deal with the extension ABI. Even those who do work on C extensions have so far been happy to work with the VC build (except for the recurrent issue of getting one's hands on the toolchain). So why should python-dev have a desire to do that work? It should be evident by now that our belief is that the large majority of Windows users is well-served by the current model (produce an official binary and a plethora of compatible extensions, which provides strong incentive to third parties to ensure that their extensions and alternative builds of core are also compatible). I think the repeated query, why isn't this model good enough for you? is being misinterpreted. I suppose that some who ask that really want to know, because if you have what they consider a good reason, they'd be willing to help. Others are asking but by you they mean the world at large, in particular, what benefit is there to the large number of users well-served by the current model? It seems python-dev would rather have us spend zero time proposing changes that allow CPython itself to be built differently than today, and all of our time on MinGW extensions. Of course they would. (Third person because I'm not competent to do the work anyway.) It's quite clear that one thing the two sides agree on is that ensuring that MinGW and VC interpreter and extension builds can mix and match is quite a bit of work. They quite naturally don't want to do that work, and don't see much point in discussing it if the (apparently) few people who need it aren't going to supply the resources. That's quite a reasonable solution, really: *both* communities avoid spending effort on mix and match, and because it's a fork, it's a different name, and people won't expect them to mix and match. I personally find 3 of the 4 combinations of how one could build CPython and how one could build extensions interesting and worth looking into for different reasons (the one that's uninteresting to me is the currently used all-MSVC method, due to its many limitations I listed earlier). Ray for example may only care about using MinGW for everything. Python's a big community with lots of room for different people to work on different aspects of the same set of problems. There's a reason we call it core. One of python-dev's more important responsibilities is to ensure that the aspects work and play well together. Aspects people tend to deprecate that responsibility (until somebody else's aspect treads on their blue suede shoes). That's as it should be, IMO -- but so is python-dev's reluctance to admit new aspects until their impact on core responsibilities is made clear. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of C compilers for Python on Windows
Stephen J. Turnbull: Sure -- as long as it works for them, though, such potential contributors don't necessarily care if it works for anybody else. My experience (in other projects) is that allowing that level of commitment to be sufficient for inclusion in the maintained code base frequently results in bug reports from third parties who try to use the new feature and discover it *doesn't* work for them. Good point. I definitely care whether patches work for everyone else. Patches should be done well and accompanied with proper documentation so new functionality is fully reproducible. If that's what's holding up review, comments in the bug threads indicating as much would be helpful. Any fork will also have to be thoroughly documented if it's to have any chance of working. Sounds good to me. You seem to think that's excessive, though: No, just hearing the words come out of my mouth they sound a little nuts. Maybe not, there are after all half a dozen or more incompatible alternate Python implementations floating around. I think most of them started as from-scratch rewrites rather than source forks, but maybe that's irrelevant. Well, for starters, most of python-dev would rather avoid any contact whatsoever with Windows. I think part of the problem is that Windows developers *of* Python are *very* rare (fingers of one hand rare). In my opinion the MSVC toolchain makes that problem worse, as it's far harder for unix developers to have any familiarity with how things work. But you do have involvement and core developers from Microsoft which is obviously incredibly important. Maybe even mandatory for Python on Windows to be viable in your eyes. Even those who do work on C extensions have so far been happy to work with the VC build (except for the recurrent issue of getting one's hands on the toolchain). It should be evident by now that our belief is that the large majority of Windows users is well-served by the current model This is not the case at all in the scientific community. NumPy and SciPy put in a lot of extra work to come up with something that is compatible with the MSVC build of CPython because they have to, not because they're happy to jump through the necessary hoops. The situation today with NumPy AIUI is they already have to build with a custom hacked MinGW toolchain that only one person knows how to use. Evidently until very recently they were using a many-years-old 32-bit-only build of GCC 3.x. Do python-dev and numpy-discussion not talk to one another? I get that not everyone uses numpy, or Windows, but it sounds like there's very little understanding, or even awareness, of the issues they face. They quite naturally don't want to do that work, and don't see much point in discussing it if the (apparently) few people who need it aren't going to supply the resources. I'm going to move the extensions with MinGW-w64 part of this conversation over to numpy-discussion, since they have a need for this today and are already using workarounds and processes that I don't think anyone is fully satisfied with. I do find this combination interesting, worth working on, and possible to make work well, but not completely in isolation as it does not address my embedding use case. but so is python-dev's reluctance to admit new aspects until their impact on core responsibilities is made clear. Okay. I'll table the discussion with python-dev for now then. -Tony ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of C compilers for Python on Windows
On 28 October 2014 14:46, Tony Kelman kel...@berkeley.edu wrote: Patches should be done well and accompanied with proper documentation so new functionality is fully reproducible. If that's what's holding up review, comments in the bug threads indicating as much would be helpful. Typically that tends to be expressed as a terse I can't get this working. That's not ideal, but is an indication. When the response is but it's easy (as it sometimes is) communication degenerates quite fast. There's an example here in this thread - it took me a *long* time, with help from a couple of people, to work out how to get a mingw toolchain I could use to try things out. Even though the premise of the whole discussion was it's easy to set up a mingw toolchain... In my opinion the MSVC toolchain makes that problem worse, as it's far harder for unix developers to have any familiarity with how things work. But you do have involvement and core developers from Microsoft which is obviously incredibly important. Maybe even mandatory for Python on Windows to be viable in your eyes. One problem I've seen a lot on other projects is that when Unix developers set up a comfortable, Unix-like environment on Windows using something like mingw, they frequently aren't aware of crucial differences between Unix and Windows and tend to write software that even though it's Windows-native, *feels* Unixy to Windows users (don't ask me to be more specific :-)). That's always going to happen, and the people with Windows experience have to take up the slack by keeping an eye out for such things, but setting the bar marginally higher, to you have to at least be willing to download and use a native Windows compiler can at least remind said Unix developers that they are in a different environment. That's *not* a criticism of anyone in the Python community, btw. Typically the experience of Windows users is very well respected by python core and package developers. But I tend to think that's partly *because* we didn't take the Unix-like toolchain approach by default. This is not the case at all in the scientific community. NumPy and SciPy put in a lot of extra work to come up with something that is compatible with the MSVC build of CPython because they have to, not because they're happy to jump through the necessary hoops. The situation today with NumPy AIUI is they already have to build with a custom hacked MinGW toolchain that only one person knows how to use. Evidently until very recently they were using a many-years-old 32-bit-only build of GCC 3.x. Do python-dev and numpy-discussion not talk to one another? I get that not everyone uses numpy, or Windows, but it sounds like there's very little understanding, or even awareness, of the issues they face. Sadly, no. The numpy developers have always been a pretty much separate world. We're seeing a bit more interaction these days, but it's very limited and far from the level that's needed. The fault (if that's the right word) probably lies on both sides. It's certainly not purely the responsibility of the core team to build communication links. As this thread has demonstrated, python-dev (and distutils-sig, which is another place that desparately needs more numpy interaction) is not the most welcoming of places for someone who is 100% focused on the scientific stack, but conversely the scientific types do tend to do their own thing, and sometimes when they do surface, they can dive in with little appreciation of the wider picture. But we can get along, and we can make progress (albeit not always as fast as either party might like). If all this thread has done is raise awareness of each others' concerns, it's still been a win IMO. I'm going to move the extensions with MinGW-w64 part of this conversation over to numpy-discussion, since they have a need for this today and are already using workarounds and processes that I don't think anyone is fully satisfied with. I do find this combination interesting, worth working on, and possible to make work well, but not completely in isolation as it does not address my embedding use case. Please keep distutils-sig in the loop as well. I can't promise we'll be able to help, but we should at least make sure we're not hindering you, and we can make you aware of when your solutions won't work outside your area of interest. Now the door is open, let's not close it again. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 475
Hi, At the end of August, I sent the PEP 475 which I wrote with Charles-François Natali: https://mail.python.org/pipermail/python-dev/2014-August/136077.html https://mail.python.org/pipermail/python-dev/2014-September/136101.html Antoine Pitrou wrote I'm +1 on the whole PEP and R. David Murray wrote Personally, I really want Python to handle EINTR for me. What's the next step? Who wants to handle this PEP? Guido? Antoine? I will try to answer to questions if previous answers were not enough. Antoine Pitrou wrote: On Unix, the ``asyncio`` module uses the wakeup file descriptor to wake up its event loop. How about Windows? I modified signal.set_wakeup_fd() in Python 3.5 to support a socket on Windows. So it becomes possible to support signals with signal.set_wakeup_fd() on Windows (for SelectorEventLoop and ProactorEventLoop): https://code.google.com/p/tulip/issues/detail?id=191 Antoine Pitrou wrote: Some signals are not interesting and should not interrupt the the application. There are two options to only interrupt an application on some signals: * Raise an exception in the signal handler, like ``KeyboardInterrupt`` for ``SIGINT`` * Use a I/O multiplexing function like ``select()`` with the Python signal wakeup file descriptor: see the function ``signal.set_wakeup_fd()``. This section looks a bit incomplete. Some calls such as os.read() or os.write() will (should) return a partial result when interrupted and they already handled 0 bytes. Perhaps other functions have a similar behaviour? In Python 3.4, os.read() is dummy: it only calls the C function read() once. With the PEP 475, read() is only called again on EINTR if the signal handler did not raise an exception. When read() returns EINTR, there is partial read, the read did not start yet. So I don't understand what should be added to the PEP. There is no specific change. Matthew Woodcraft wrote: In any case I think PEP 475 should be explaining what is going to happen to signal.siginterrupt(). Will setting flag=True be supported? If so, will doing so change the behaviour of those parts of the stdlib which have already been modified to retry after EINTR? In Python 3.4, signal.signal() calls siginterrupt(signum, True): syscalls raise InterruptedError when interrupted by a signal. Calling explicitly signal.siginterrupt(signum, True) doesn't change anything. In Python 3.4, signal.siginterrupt(signum, False) reduces the cases when InterruptedError is raised, but they are still cases when InterruptedError is raised. The exact behaviour probably depends on the operating system or even the version of the operating system. It's better to not rely on siginterrupt(False) to write portable code in Python 3.4. With the PEP, signal.siginterrupt(signum, False) is still supported. The PEP doesn't change the behaviour when the syscall is directly restarted by the C library. If the function returns EINTR, the interrupted syscall is retried if the signal handler didn't raise an exception. The main problem of siginterrupt(False) is that the Python signal handler is *not* called if the C library directly retries the interrupted syscall. Note: if signals are blocked, the C signal handlers are not called, so the PEP doesn't change the behaviour neither. Victor wrote: I think that the stdlib should not handle InterruptedError exception anymore in the Python code, to simplify the code. I modified the PEP to mention that: https://hg.python.org/peps/rev/627fefe0394f Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 475
Oh, I forgot the link to the PEP: http://legacy.python.org/dev/peps/pep-0475/ Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of C compilers for Python on Windows
On 10/28/2014 6:45 AM, Stephen J. Turnbull wrote: because it's a fork, it's a different name I think this is an important point, and first brought to this discussion here. A fork is _not_ called Python, but something else... but if it is kept extremely compatible and up-to-date in the hopes of reintegration once it proves that it solves a problem, and that there are sufficient users, then such hopes seem reasonable. And targeting the new future compatible MSVCRT sounds like the right approach, although it won't solve today's problems today, but it may solve tomorrow's problems for a long time into the future... if the MinGW people can be convinced to support that new MSVCRT as well. In addition to all the components that are enabled by MinGW (particularly Fortran based extensions), one must remember that the current Windows Python also has extensions that interface to MSVC libraries that have never been ported to MinGW or Linux, and may never be. So an incompatible MinGW-built fork will lose some of those extensions... they may not be important to the folks that need MinGW, but that would be where why the community would be split if the MinGW fork is not compatible with (some version of MSVC). Of course, the current MSVC-based community is _already_ having issues with incompatible versions of MSVC (not limiting that community to Python, but broader users of MSVC)... enough problems that even M$ has noticed that their incompatibilities are problematical, and are attempting to address... not just for Python, but for many other systems and libraries as well. So gathering around and supporting their attempts to achieve that, by using their new system early, when feedback can still have a chance of improving that new future compatible system, will benefit everyone... but that time is limited, from what Steve Dower reports of the schedule... hoping to be ready for Python 3.5. So now is an excellent time for this discussion to be happening, and if some work can be done to fork, pull the patches together, and tweak them to work with the new MSVC, in the Python 3.5 timeframe, you can have a phenomenal result, even if it is still a fork at that time. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 475
I would like this to happen, but I'm afraid of breakage, and I don't have time. I would be okay if Antoine agrees to be the PEP-BDFL. On Tue, Oct 28, 2014 at 2:13 PM, Victor Stinner victor.stin...@gmail.com wrote: Oh, I forgot the link to the PEP: http://legacy.python.org/dev/peps/pep-0475/ Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 475
On Tue, 28 Oct 2014 22:07:45 +0100 Victor Stinner victor.stin...@gmail.com wrote: Hi, At the end of August, I sent the PEP 475 which I wrote with Charles-François Natali: https://mail.python.org/pipermail/python-dev/2014-August/136077.html https://mail.python.org/pipermail/python-dev/2014-September/136101.html Antoine Pitrou wrote I'm +1 on the whole PEP and R. David Murray wrote Personally, I really want Python to handle EINTR for me. What's the next step? Who wants to handle this PEP? Guido? Antoine? Is there an implementation somewhere? (I can handle the PEP if Guido doesn't want to and other people agree) Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 475
2014-10-28 22:36 GMT+01:00 Antoine Pitrou solip...@pitrou.net: Is there an implementation somewhere? There is no implementation yet. This time, I chose to focus on the PEP before working on an implementation :-) We can work on the implementation if it helps discuss the PEP. I created a repository 3 months ago, but it has no commit yet: https://hg.python.org/features/eintr/ The issue contains a first patch: http://bugs.python.org/issue18885 Antoine wrote: (I can handle the PEP if Guido doesn't want to and other people agree) Guido just wrote: I would be okay if Antoine agrees to be the PEP-BDFL. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of C compilers for Python on Windows
Tony Kelman writes: No, just hearing the words come out of my mouth they sound a little nuts. Maybe not, there are after all half a dozen or more incompatible alternate Python implementations floating around. I think most of them started as from-scratch rewrites rather than source forks, but maybe that's irrelevant. Well, they have different names and they clearly are not intended to be ABI compatible, so noone expects that. OTOH, there clearly is an expectation among many (and not just in the Windows world, cf. all of the distros that provide whole stacks of everything for each version of Python) that downloaded packages will just work without incompatibility. Well, for starters, most of python-dev would rather avoid any contact whatsoever with Windows. I think part of the problem is that Windows developers *of* Python are *very* rare (fingers of one hand rare). In my opinion the MSVC toolchain makes that problem worse, as it's far harder for unix developers to have any familiarity with how things work. I've used Cygwin, I've used MinGW, and I've used VC. Sure, the former two are GCC-based so I have a lot of muscle memory for command-line switches. But that's not very important; the pain of using Windows is what drives me away from all of them. But you do have involvement and core developers from Microsoft which is obviously incredibly important. Maybe even mandatory for Python on Windows to be viable in your eyes. No, I don't think that's true. What I think *is* true is that most developers on Windows do have access to Microsoft tools, so we do need to provide compatibility with them. As you say, the VC toolchain is not all things to all men, but what's visible to python-dev makes it more important than Cygwin or MinGW. See Paul Moore's post about communications between the scientific Python community and python-dev for what I mean by visible. It should be evident by now that our belief is that the large majority of Windows users is well-served by the current model This is not the case at all in the scientific community. NumPy and SciPy put in a lot of extra work to come up with something that is compatible with the MSVC build of CPython because they have to, not because they're happy to jump through the necessary hoops. Agreed. This is well-known to python-dev, and AFAICS it *is* a concern for us. However, as Paul points out, a bridge needs to be built. Your posts have been a contribution to building that bridge, for sure, but more work on the bridge is needed. Do python-dev and numpy-discussion not talk to one another? Exactly the issue here. To resolve this, we need to talk more. Unfortunately, I'm not one to help build the bridge as I haven't developed on Windows at all since about 2003. I'm going to move the extensions with MinGW-w64 part of this conversation over to numpy-discussion, As far as I can tell, that's a good idea right now. They have the need, they have the expertise, both of which are somewhat lacking here. Okay. I'll table the discussion with python-dev for now then. I hope you'll be able to come pick it back up at some point. You might want to interact with Steve Dower off-list, as he's spearheading the effort to move the official builds to the stable ABI version of MSVC. Once that's in place, the MinGW guys will have a stationary target which is up to date to shoot at. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
