Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR
In article CAMpsgwabYhXB0OG3UhdX=fucyonajgzpwd-g8stdaukjzpj...@mail.gmail.com, Victor Stinner victor.stin...@gmail.com wrote: 2014-09-02 23:03 GMT+02:00 Matthew Woodcraft matt...@woodcraft.me.uk: In any case I think PEP 475 should be explaining what is going to happen to signal.siginterrupt(). Will setting flag=True be supported? I first proposed to deprecate the function, but Charles-François thinks that it's unrelated to the PEP (it can be addressed later). The function will still be available and work. If so, will doing so change the behaviour of those parts of the stdlib which have already been modified to retry after EINTR? I think that the stdlib should not handle InterruptedError exception anymore in the Python code, to simplify the code. That seems good to me. I think it's worth writing in the PEP. But if Python is going to provide its own higher-level model of signal handling, then I think signal.siginterrupt() will need to be documented in terms of that model; it should be saying something along the lines of if a signal arrives while Python is blocking in a system call then InterruptedError will be raised, and I suppose it will need to say what happens if the signal handler also raised an exception. (I think it would be helpful if we could tell people if you want the old EINTR behaviour, just do this simple thing. And I suppose siginterrupt flag=True is a candidate for that.) Why do you want the old behaviour? Purely to avoid breaking existing programs, particularly in ways which will require effort to fix. I think Python's backward-compatibility rules are basically the behavior of an API must not change without going through the deprecation process. If we're going to be making an exception to that here, then it would be much better to say here's a simple change to make to keep your old program working, rather than please rewrite your fiddly signal-handling code to use more modern techniques, however much nicer those modern techniques are. Or perhaps it would be better to not make an exception at all, and instead add an 'interrupt_syscalls' boolean keyword argument to signal.signal(), while strongly recommending that people set it False. -M- ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR
On 2 September 2014 07:17, Matthew Woodcraft matt...@woodcraft.me.uk wrote: (The program handles SIGTERM so that it can do a bit of cleanup before exiting, and it uses the signal-handler-sets-a-flag technique. The call that might be interrupted is sleep(), so the program doesn't strictly _rely_ on the existing behaviour; it would just become very slow to exit.) Making an exception for sleep() (i.e. still letting it throw EINTR) sounds like a reasonable idea to me. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR
On Mon, 1 Sep 2014 21:17:33 + (UTC) Matthew Woodcraft matt...@woodcraft.me.uk wrote: If such applications exist, they are not portable and are subject to race conditions (deadlock if the signal comes before the system call). The program is certainly not portable (which is not any kind of a problem), and as pselect is unavailable there is indeed the usual theoretical race (which has not been a problem in practice in the ten years it's been running). (The program handles SIGTERM so that it can do a bit of cleanup before exiting, and it uses the signal-handler-sets-a-flag technique. The call that might be interrupted is sleep(), so the program doesn't strictly _rely_ on the existing behaviour; it would just become very slow to exit.) So, if it's just for process exit, just let the signal handler raise an exception and catch the exception at the top-level. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR
Nick Coghlan ncogh...@gmail.com wrote: On 2 September 2014 07:17, Matthew Woodcraft matt...@woodcraft.me.uk wrote: (The program handles SIGTERM so that it can do a bit of cleanup before exiting, and it uses the signal-handler-sets-a-flag technique. The call that might be interrupted is sleep(), so the program doesn't strictly _rely_ on the existing behaviour; it would just become very slow to exit.) Making an exception for sleep() (i.e. still letting it throw EINTR) sounds like a reasonable idea to me. I think people who use sleep() in their programs could benefit from not having to worry about EINTR as much as anyone else. And I don't think there's any reason to suppose that programs using sleep() are more likely than others to want to see EINTR. I think blocking network operations are more likely to be involved. -M- ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR
Antoine Pitrou solip...@pitrou.net wrote: Matthew Woodcraft matt...@woodcraft.me.uk wrote: (The program handles SIGTERM so that it can do a bit of cleanup before exiting, and it uses the signal-handler-sets-a-flag technique. The call that might be interrupted is sleep(), so the program doesn't strictly _rely_ on the existing behaviour; it would just become very slow to exit.) So, if it's just for process exit, just let the signal handler raise an exception and catch the exception at the top-level. I wouldn't recommend that anyone take this advice. The signal might come at some time other than the sleep(), and introducing an exception which can mysteriously appear at abitrary points is not a way to make life easier. Nonetheless I'm sure my program would be easy enough to fix to use set_wakeup_fd() or signalfd() if necessary. I'm not saying we shouldn't make this backwards-incompatible change because it will be hard to update existing programs to cope; I'm saying we shouldn't pretend it doesn't count as a backwards-incompatible change. In any case I think PEP 475 should be explaining what is going to happen to signal.siginterrupt(). Will setting flag=True be supported? If so, will doing so change the behaviour of those parts of the stdlib which have already been modified to retry after EINTR? (I think it would be helpful if we could tell people if you want the old EINTR behaviour, just do this simple thing. And I suppose siginterrupt flag=True is a candidate for that.) -M- ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR
2014-09-02 23:02 GMT+02:00 Matthew Woodcraft matt...@woodcraft.me.uk: I think people who use sleep() in their programs could benefit from not having to worry about EINTR as much as anyone else. The behaviour of time.sleep() is worse than what I expected. On UNIX, if select() fails with EINTR, time.sleep() calls PyErr_CheckSignals(). If the signal handler doesn't raise an exception, time.sleep() returns None and just simply ignores the error. But on Windows, it's the opposite. If time.sleep() is interrupt by CTRL+c, time.sleep() raises an InterruptedError... Good luck to write portable code :-p With the PEP 475, time.sleep(secs) now has a well defined behaviour. It sleeps at least secs seconds, retry the syscall on EINTR, and raises an exception if the signal handler raises an exception. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR
2014-09-02 23:03 GMT+02:00 Matthew Woodcraft matt...@woodcraft.me.uk: In any case I think PEP 475 should be explaining what is going to happen to signal.siginterrupt(). Will setting flag=True be supported? I first proposed to deprecate the function, but Charles-François thinks that it's unrelated to the PEP (it can be addressed later). The function will still be available and work. If so, will doing so change the behaviour of those parts of the stdlib which have already been modified to retry after EINTR? I think that the stdlib should not handle InterruptedError exception anymore in the Python code, to simplify the code. (I think it would be helpful if we could tell people if you want the old EINTR behaviour, just do this simple thing. And I suppose siginterrupt flag=True is a candidate for that.) Why do you want the old behaviour? Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR
On 31 August 2014 22:38, Victor Stinner victor.stin...@gmail.com wrote: This case is described as the use case #2 in the PEP, so it is supported. As written in the PEP, if you want to be notified of the signal, set a signal handler which raises an exception. For example the default signal handler for SIGINT raises KeyboardInterrupt. Wait - sigint? Does this mean that (unless the application adds a signal handler) keyboard interrupts will be in effect ignored while in a system call? I'm not sure I like that - I'd rather Ctrl-C always interrupted the program. Specifically, in one-off scripts that *don't* take care to handle all errors appropriately and just show the traceback... Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR
Victor Stinner wrote: Le 1 sept. 2014 00:17, Marko Rauhamaa ma...@pacujo.net mailto:ma...@pacujo.net a écrit : If a signal is received when read() or write() has completed its task partially ( 0 bytes), no EINTR is returned but the partial count. Obviously, Python should take that possibility into account so that raising an exception in the signal handler (as mandated by the PEP) doesn't cause the partial result to be lost on os.read() or os.write(). This case is unrelated to the PEP, the PEP only changes the behaviour when a syscall fails with EINTR. I think there's a problem here, though. As thing stand, a signal handler that doesn't raise an exception can set a flag, and code after the read() can test it. Under the proposed scheme, the signal handler has to be made to raise an exception so that the read will be broken out of in the EINTR case. But what happens if the read returns *without* an EINTR? The signal handler will still raise an exception, which is either going to clobber the partial return value or mess up the code that does something with it. -- Greg ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR
No, it's the opposite. The PEP doesn't change the default behaviour of SIGINT: CTRL+C always interrupt the program. Victor Le 1 sept. 2014 08:12, Paul Moore p.f.mo...@gmail.com a écrit : On 31 August 2014 22:38, Victor Stinner victor.stin...@gmail.com wrote: This case is described as the use case #2 in the PEP, so it is supported. As written in the PEP, if you want to be notified of the signal, set a signal handler which raises an exception. For example the default signal handler for SIGINT raises KeyboardInterrupt. Wait - sigint? Does this mean that (unless the application adds a signal handler) keyboard interrupts will be in effect ignored while in a system call? I'm not sure I like that - I'd rather Ctrl-C always interrupted the program. Specifically, in one-off scripts that *don't* take care to handle all errors appropriately and just show the traceback... Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR
Le 1 sept. 2014 02:40, Greg Ewing greg.ew...@canterbury.ac.nz a écrit : Victor Stinner wrote: As written in the PEP, if you want to be notified of the signal, set a signal handler which raises an exception. I'm not convinced that this covers all possible use cases. It might be all right if you have control over the signal handler, but what if you don't? Ok, let's say that a syscall is interrupted by a signal, but rhe signal doesn't raise an exception. So your program can only be interrupted if the signal is received during a syscall, right? I don't think that such program is reliable. Python should not promote such design. It should behave the same if the signal is received during a CPU-bound function. Extract of the PEP: Backward Compatibility: Applications relying on the fact that system calls are interruptedwith ``InterruptedError`` will hang. The authors of this PEP don'tthink that such application exist.If such applications exist, they are not portable and are subject torace conditions (deadlock if the signal comes before the system call).These applications must be fixed to handle signals differently, tohave a reliable behaviour on all platforms and all Python versions.For example, use a signal handler which raises an exception, or use awakeup file descriptor.For applications using event loops, ``signal.set_wakeup_fd()`` is therecommanded option to handle signals. The signal handler writes signalnumbers into the file descriptor and the event loop is awaken to readthem. The event loop can handle these signals without the restrictionof signal handlers. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR
Victor Stinner victor.stin...@gmail.com: No, it's the opposite. The PEP doesn't change the default behaviour of SIGINT: CTRL+C always interrupt the program. Which raises an interesting question: what happens to the os.read() return value if SIGINT is received? Marko ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR
There's no return value, a KeywordInterrupt exception is raised. The PEP wouldn't change this behavior. As for the general behavior: all programming languages/platforms handle EINTR transparently. It's high time for Python to have a sensible behavior in this regard. 2014-09-01 8:38 GMT+01:00 Marko Rauhamaa ma...@pacujo.net: Victor Stinner victor.stin...@gmail.com: No, it's the opposite. The PEP doesn't change the default behaviour of SIGINT: CTRL+C always interrupt the program. Which raises an interesting question: what happens to the os.read() return value if SIGINT is received? Marko ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/cf.natali%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR
Charles-François Natali cf.nat...@gmail.com: Which raises an interesting question: what happens to the os.read() return value if SIGINT is received? There's no return value, a KeywordInterrupt exception is raised. The PEP wouldn't change this behavior. Slightly disconcerting... but I'm sure overriding SIGINT would cure that. You don't want to lose data if you want to continue running. As for the general behavior: all programming languages/platforms handle EINTR transparently. C doesn't. EINTR is there for a purpose. I sure hope Python won't bury it under opaque APIs. The two requirements are: * Allow the application to react to signals immediately in the main flow. * Don't lose information. Marko ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR
2014-09-01 12:15 GMT+01:00 Marko Rauhamaa ma...@pacujo.net: Charles-François Natali cf.nat...@gmail.com: Which raises an interesting question: what happens to the os.read() return value if SIGINT is received? There's no return value, a KeywordInterrupt exception is raised. The PEP wouldn't change this behavior. Slightly disconcerting... but I'm sure overriding SIGINT would cure that. You don't want to lose data if you want to continue running. As for the general behavior: all programming languages/platforms handle EINTR transparently. C doesn't. EINTR is there for a purpose. Python is slightly higher level than C, right? I was referring to Java, go, Haskell... Furthermore, that's not true: many operating systems actually restart syscalls by default (including Linux, man 7 signal): Interruption of system calls and library functions by signal handlers If a signal handler is invoked while a system call or library function call is blocked, then either: * the call is automatically restarted after the signal handler returns; or * the call fails with the error EINTR. Which of these two behaviors occurs depends on the interface and whether or not the signal handler was established using the SA_RESTART flag (see sigaction(2)). The details vary across UNIX systems; below, the details for Linux. The reason the interpreter is subject to so many EINTR is that we *explicitly* clear SA_RESTART because the C-level signal handler must be handled by the interpreter to have a chance to run the Python-level handlers from the main loop. There are many aspects of signal handling in Python that make it different from C: if you want C semantics, stick to C. I do not want to have to put all blocking syscalls within a try/except loop: have a look at the stdlib code, you'll see it's really a pain and ugly. And look at the number of EINTR-related syscalls we've had. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR
On Mon, 01 Sep 2014 19:15:33 +1200 Greg Ewing greg.ew...@canterbury.ac.nz wrote: Victor Stinner wrote: Le 1 sept. 2014 00:17, Marko Rauhamaa ma...@pacujo.net mailto:ma...@pacujo.net a écrit : If a signal is received when read() or write() has completed its task partially ( 0 bytes), no EINTR is returned but the partial count. Obviously, Python should take that possibility into account so that raising an exception in the signal handler (as mandated by the PEP) doesn't cause the partial result to be lost on os.read() or os.write(). This case is unrelated to the PEP, the PEP only changes the behaviour when a syscall fails with EINTR. I think there's a problem here, though. As thing stand, a signal handler that doesn't raise an exception can set a flag, and code after the read() can test it. Under the proposed scheme, the signal handler has to be made to raise an exception so that the read will be broken out of in the EINTR case. The PEP already addresses this remark: Applications relying on the fact that system calls are interrupted with ``InterruptedError`` will hang. The authors of this PEP don't think that such application exist. If such applications exist, they are not portable and are subject to race conditions (deadlock if the signal comes before the system call). Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR
Hi, I'm +1 on the whole PEP. Writing a signal handler is difficult, only async-signal safe functions can be called. You mean a C signal handler? Python signal handlers are not restricted. Some signals are not interesting and should not interrupt the the application. There are two options to only interrupt an application on some signals: * Raise an exception in the signal handler, like ``KeyboardInterrupt`` for ``SIGINT`` * Use a I/O multiplexing function like ``select()`` with the Python signal wakeup file descriptor: see the function ``signal.set_wakeup_fd()``. This section looks a bit incomplete. Some calls such as os.read() or os.write() will (should) return a partial result when interrupted and they already handled 0 bytes. Perhaps other functions have a similar behaviour? On Unix, the ``asyncio`` module uses the wakeup file descriptor to wake up its event loop. How about Windows? Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR
On Mon, 01 Sep 2014 08:30:27 +0300, Marko Rauhamaa ma...@pacujo.net wrote: R. David Murray rdmur...@bitdance.com: PS: I recently switched from using selectors to using a timeout on a socket because in that particular application I could, and because reading a socket with a timeout handles EINTR (in recent python versions), whereas reading a non-blocking socket doesn't. Under the hood, a socket with a timeout is a non-blocking socket. Under what circumstances would a nonblocking socket generate an EINTR? Windows. Enough said? The exact error message was: BlockingIOError on my non-blocking socket: 'a non-blocking socket operation could not be completed immediately Needless to say, I was not expecting this, and was about to tear my remaining hair out about having to completely restructure the code in order to be able to handle an EINTR on a read on an FD that I got back from select as ready, until I realized that the way the code had evolved the only thing I still needed the select for was the timeout, and that the EINTR bug in sockets with a timeout had already been fixed (thank goodness I'm able to use python3.4 for this project). I got lucky, but this is clearly a serious problem for writing selectors based code on Windows. This should tell you just about everything you need to know about why we want to fix this problem so that things work cross platform. --David ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR
On Mon, 01 Sep 2014 14:15:52 +0300, Marko Rauhamaa ma...@pacujo.net wrote: Charles-François Natali cf.nat...@gmail.com: Which raises an interesting question: what happens to the os.read() return value if SIGINT is received? There's no return value, a KeywordInterrupt exception is raised. The PEP wouldn't change this behavior. Slightly disconcerting... but I'm sure overriding SIGINT would cure that. You don't want to lose data if you want to continue running. As for the general behavior: all programming languages/platforms handle EINTR transparently. C doesn't. EINTR is there for a purpose. I sure hope Python won't bury it under opaque APIs. The two requirements are: * Allow the application to react to signals immediately in the main flow. You don't want to be writing your code in Python then. In Python you *never* get to react immediately to signals. The interpreter sets a flag and calls the python signal handler later. Yes, the call is ASAP, but ASAP is *not* immediately. * Don't lose information. Marko ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/rdmurray%40bitdance.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR
On Mon, 01 Sep 2014 11:47:07 -0400 R. David Murray rdmur...@bitdance.com wrote: The two requirements are: * Allow the application to react to signals immediately in the main flow. You don't want to be writing your code in Python then. In Python you *never* get to react immediately to signals. The interpreter sets a flag and calls the python signal handler later. Yes, the call is ASAP, but ASAP is *not* immediately. Especially if the signal is delivered to another thread (which is OS-dependent), and the main thread is blocked in *another* system call ;-) Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR
R. David Murray rdmur...@bitdance.com: Windows. Enough said? [...] This should tell you just about everything you need to know about why we want to fix this problem so that things work cross platform. I feel your pain. Well, not really; I just don't want my linux bliss to be taken away. Marko ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR
R. David Murray rdmur...@bitdance.com: On Mon, 01 Sep 2014 14:15:52 +0300, Marko Rauhamaa ma...@pacujo.net wrote: * Allow the application to react to signals immediately in the main flow. You don't want to be writing your code in Python then. In Python you *never* get to react immediately to signals. The interpreter sets a flag and calls the python signal handler later. Yes, the call is ASAP, but ASAP is *not* immediately. You don't have to get that philosophical. Immediately means, without delay, without further I/O. Marko ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR
Victor Stinner victor.stin...@gmail.com wrote: HTML version: http://legacy.python.org/dev/peps/pep-0475/ PEP: 475 Title: Retry system calls failing with EINTR I think the proposed design for how Python should behave is a good one. But I think this proposal needs to be treated in the same way as any other backwards-incompatible change. Applications relying on the fact that system calls are interrupted with ``InterruptedError`` will hang. The authors of this PEP don't think that such application exist. The authors are mistaken here. I have a program still running which was designed around this behaviour. My company won't be inconvenienced by this change because I can't imagine the elderly program ever being ported to Python 3. But I think it's very likely there are other such programs out there. If such applications exist, they are not portable and are subject to race conditions (deadlock if the signal comes before the system call). The program is certainly not portable (which is not any kind of a problem), and as pselect is unavailable there is indeed the usual theoretical race (which has not been a problem in practice in the ten years it's been running). (The program handles SIGTERM so that it can do a bit of cleanup before exiting, and it uses the signal-handler-sets-a-flag technique. The call that might be interrupted is sleep(), so the program doesn't strictly _rely_ on the existing behaviour; it would just become very slow to exit.) -M- ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR
Victor Stinner victor.stin...@gmail.com: Proposition === If a system call fails with ``EINTR``, Python must call signal handlers: call ``PyErr_CheckSignals()``. If a signal handler raises an exception, the Python function fails with the exception. Otherwise, the system call is retried. If the system call takes a timeout parameter, the timeout is recomputed. Signals are tricky and easy to get wrong, to be sure, but I think it is dangerous for Python to unconditionally commandeer signal handling. If the proposition is accepted, there should be a way to opt out. Marko ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR
Hi, Sorry but I don't understand your remark. What is your problem with retrying syscall on EINTR? Can you please elaborate? What do you mean by get wrong? Victor Le dimanche 31 août 2014, Marko Rauhamaa ma...@pacujo.net a écrit : Victor Stinner victor.stin...@gmail.com javascript:;: Proposition === If a system call fails with ``EINTR``, Python must call signal handlers: call ``PyErr_CheckSignals()``. If a signal handler raises an exception, the Python function fails with the exception. Otherwise, the system call is retried. If the system call takes a timeout parameter, the timeout is recomputed. Signals are tricky and easy to get wrong, to be sure, but I think it is dangerous for Python to unconditionally commandeer signal handling. If the proposition is accepted, there should be a way to opt out. Marko ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR
Victor Stinner victor.stin...@gmail.com: Sorry but I don't understand your remark. What is your problem with retrying syscall on EINTR? The application will often want the EINTR return (exception) instead of having the function resume on its own. Can you please elaborate? What do you mean by get wrong? Proper handling of signals is difficult and at times even impossible. For example it is impossible to wake up reliably from the select(2) system call when a signal is generated (which is why linux now has pselect). Marko ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR
On 08/31/2014 02:19 PM, Marko Rauhamaa wrote: Victor Stinner victor.stin...@gmail.com: Sorry but I don't understand your remark. What is your problem with retrying syscall on EINTR? The application will often want the EINTR return (exception) instead of having the function resume on its own. Examples? As an ignorant person in this area, I do not know why I would ever want to have EINTR raised instead just getting the results of, say, my read() call. -- ~Ethan~ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR
Le dimanche 31 août 2014, Marko Rauhamaa ma...@pacujo.net a écrit : Victor Stinner victor.stin...@gmail.com javascript:;: Sorry but I don't understand your remark. What is your problem with retrying syscall on EINTR? The application will often want the EINTR return (exception) instead of having the function resume on its own. This case is described as the use case #2 in the PEP, so it is supported. As written in the PEP, if you want to be notified of the signal, set a signal handler which raises an exception. For example the default signal handler for SIGINT raises KeyboardInterrupt. Can you please elaborate? What do you mean by get wrong? Proper handling of signals is difficult and at times even impossible. For example it is impossible to wake up reliably from the select(2) system call when a signal is generated (which is why linux now has pselect). In my experience, using signal.set_wakeup_fd() works well with select(), even on Windows. The PEP promotes this. It is even thread safe. I don't know issues of signals with select() (and without a file descriptor used to wake up it). Python now exposes signal.pthread_sigmask(), I don't know if it helps. In my experience, signals don't play well with multithreading. On FreeBSD, the signal is send to a random thread. So you must have the same signal mask on all threads if you want to rely on them. But I don't get you point. How does this PEP make the situation worse? Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR
Victor Stinner victor.stin...@gmail.com: But I don't get you point. How does this PEP make the situation worse? Did I say it would? I just wanted to make sure the system call resumption doesn't become mandatory. Haven't thought through what the exception raising technique would entail. It might be perfectly ok apart from being a change to the signal handler API. I don't know issues of signals with select() (and without a file descriptor used to wake up it). A signal handler often sets a flag, which is inspected when select() returns. The problem is when a signal arrives between testing the flag and calling select(). The pselect() system call allows you to block signals and have the system call unblock them correctly to avoid the race. Marko ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR
Ethan Furman et...@stoneleaf.us: On 08/31/2014 02:19 PM, Marko Rauhamaa wrote: The application will often want the EINTR return (exception) instead of having the function resume on its own. Examples? As an ignorant person in this area, I do not know why I would ever want to have EINTR raised instead just getting the results of, say, my read() call. Say you are writing data into a file and it takes a long time (because there is a lot of data or the medium is very slow or there is a hardware problem). You might have designed in a signaling scheme to address just this possibility. Then, the system call had better come out right away without trying to complete the full extent of the call. If a signal is received when read() or write() has completed its task partially ( 0 bytes), no EINTR is returned but the partial count. Obviously, Python should take that possibility into account so that raising an exception in the signal handler (as mandated by the PEP) doesn't cause the partial result to be lost on os.read() or os.write(). Marko ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR
Le 1 sept. 2014 00:04, Marko Rauhamaa ma...@pacujo.net a écrit : Victor Stinner victor.stin...@gmail.com: But I don't get you point. How does this PEP make the situation worse? Did I say it would? I just wanted to make sure the system call resumption doesn't become mandatory. The syscall is only retried on EINTR if the signal handler didn't raise an exception. So it is not always retried: Proposition If a system call fails with ``EINTR``, Python must call signalhandlers: call ``PyErr_CheckSignals()``. If a signal handler raisesan exception, the Python function fails with the exception.Otherwise, the system call is retried. If the system call takes atimeout parameter, the timeout is recomputed. Haven't thought through what the exception raising technique would entail. It might be perfectly ok apart from being a change to the signal handler API. I don't think that it is safe to expect an InterruptedError if the signal handler doesn't raise an exception. Many Python module already retry the syscall on EINTR. So I'm not sure that the PEP is really a major change. It's just to make Python more homogeneous, always have the same reliable and portable behaviour. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR
On Mon, 01 Sep 2014 01:15:12 +0300 Marko Rauhamaa ma...@pacujo.net wrote: If a signal is received when read() or write() has completed its task partially ( 0 bytes), no EINTR is returned but the partial count. Obviously, Python should take that possibility into account so that raising an exception in the signal handler (as mandated by the PEP) doesn't cause the partial result to be lost on os.read() or os.write(). If the signal handler is called, the exception *will* be raised. There's no guarantee at which point in the Python code it will be raised (it's implementation-dependent), but it's near impossible to protect regular Python code against such asynchronous exceptions. Which is why you should switch to a wakeup fd scheme as mentioned by Victor, if you want to rely on signals at all. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR
Le 1 sept. 2014 00:17, Marko Rauhamaa ma...@pacujo.net a écrit : If a signal is received when read() or write() has completed its task partially ( 0 bytes), no EINTR is returned but the partial count. Obviously, Python should take that possibility into account so that raising an exception in the signal handler (as mandated by the PEP) doesn't cause the partial result to be lost on os.read() or os.write(). This case is unrelated to the PEP, the PEP only changes the behaviour when a syscall fails with EINTR. (When Python gets a signal, the C signal handler is immediatly called. The handler sets a flag which is cheched before executing an instruction. The Python signal handler can be called between two Python instructions. In some cases, it may be called earlier in functions checking manually the flag. IMO the exact behaviour is undefined. Python tries to call the Python signal handler as soon as possible, with a low performance overhead.) Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR
Victor Stinner wrote: As written in the PEP, if you want to be notified of the signal, set a signal handler which raises an exception. I'm not convinced that this covers all possible use cases. It might be all right if you have control over the signal handler, but what if you don't? I think it's best if the functions in the os module remain thin wrappers that expose the OS functionality as fully and directly as possible. Anything else should be provided by higher-level facilities. -- Greg ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR
On Sun, Aug 31, 2014 at 3:28 PM, Greg Ewing greg.ew...@canterbury.ac.nz wrote: Victor Stinner wrote: As written in the PEP, if you want to be notified of the signal, set a signal handler which raises an exception. I'm not convinced that this covers all possible use cases. It might be all right if you have control over the signal handler, but what if you don't? I think it's best if the functions in the os module remain thin wrappers that expose the OS functionality as fully and directly as possible. Anything else should be provided by higher-level facilities. I'm inclined to agree about keeping the os module thin. If we were to recreate Python today, from scratch, it might make sense to hide this by default, but now there's almost certainly code out there that depends on the current behavior. But I also agree that it's hard to pin down which higher level Python library calls are going to be using system calls. My http://stromberg.dnsalias.org/~strombrg/pypty/ program had a problem with window resizing for a while (SIGWINCH), and although I use it pretty much daily now without problems, I'm still not sure I got 100% of the possibilities covered. Fortunately, wrapping a system call can be as simple as: def retry_on_eintr(function, *args, **kw): ''' Retry a system call until one of the following happens: 1) It succeeds 2) It errors with something other than EINTR ''' while True: try: return function(*args, **kw) except OSError: nothing, extra, nothing2 = sys.exc_info() dummy = nothing dummy = nothing2 if extra.errno == errno.EINTR: continue else: raise Note that os.read() and os.write() need different handling. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR
On Sun, 31 Aug 2014 20:14:50 -0700, Dan Stromberg drsali...@gmail.com wrote: On Sun, Aug 31, 2014 at 3:28 PM, Greg Ewing greg.ew...@canterbury.ac.nz wrote: Victor Stinner wrote: As written in the PEP, if you want to be notified of the signal, set a signal handler which raises an exception. I'm not convinced that this covers all possible use cases. It might be all right if you have control over the signal handler, but what if you don't? I think it's best if the functions in the os module remain thin wrappers that expose the OS functionality as fully and directly as possible. Anything else should be provided by higher-level facilities. I'm inclined to agree about keeping the os module thin. If we were to recreate Python today, from scratch, it might make sense to hide this by default, but now there's almost certainly code out there that depends on the current behavior. But I also agree that it's hard to pin down which higher level Python library calls are going to be using system calls. My http://stromberg.dnsalias.org/~strombrg/pypty/ program had a problem with window resizing for a while (SIGWINCH), and although I use it pretty much daily now without problems, I'm still not sure I got 100% of the possibilities covered. Fortunately, wrapping a system call can be as simple as: def retry_on_eintr(function, *args, **kw): ''' Retry a system call until one of the following happens: 1) It succeeds 2) It errors with something other than EINTR ''' while True: try: return function(*args, **kw) except OSError: nothing, extra, nothing2 = sys.exc_info() dummy = nothing dummy = nothing2 if extra.errno == errno.EINTR: continue else: raise Note that os.read() and os.write() need different handling. Personally, I really want Python to handle EINTR for me. And indeed, that has been what we have been doing for a while now, piece by piece, bug by bug. Victor just wants to systematize and document that, and I think that's a good idea. We've been consistently treating lack of handling of EINTR as a bug. If there are *real* cases where that causes a backward compatibility problem, then we need to know. But so far, we have gotten zero complaints about the cases that we have fixed. --David PS: I recently switched from using selectors to using a timeout on a socket because in that particular application I could, and because reading a socket with a timeout handles EINTR (in recent python versions), whereas reading a non-blocking socket doesn't. Under the hood, a socket with a timeout is a non-blocking socket. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR
R. David Murray rdmur...@bitdance.com: PS: I recently switched from using selectors to using a timeout on a socket because in that particular application I could, and because reading a socket with a timeout handles EINTR (in recent python versions), whereas reading a non-blocking socket doesn't. Under the hood, a socket with a timeout is a non-blocking socket. Under what circumstances would a nonblocking socket generate an EINTR? I believe the biggest EINTR problem child is file I/O, which is always blocking in linux. Marko ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
