Re: [Tutor] UPDATE: Is there a 'hook' to capture all exits from a python program?
Alan Gauld wrote: On 20/03/15 09:37, Peter Otten wrote: def close_relay(e=None,v=None,t=None): try: if not relay_closed() really_close_relay() except: really_close_relay() The purpose of the if clause is to ensure that if the function is called many times you only close the relay once (I surmised that more than once could be harmful?) import sys, atexit atexit.register(close_relay) sys.excepthook = close_relay atexit should be overkill, but it might be needed if for some reason the interpreter dies while performing the usual cleanup. excepthook replaces the usual exception mechanism with a clean up. This is needed for cases where an exception occurs before getting to the finally. We want to close the relay ASAP, not waiting till the interpreter decides to call the finally. try: main program here finally: close_relay() This is the happy path where everything shuts down as expected. That reeks of cargo cult. Are there actual scenarios for each of the three mechanisms where it is the only one that works? In real-time you never trust anything. Always cover your back. I would expect that try: main program here finally: close_relay() provides the same level of confidence, Only if the interpreter is behaving as normal. The hooks are to try (we hope) to catch cases where the interpreter has broken its normal flow. So the scenarios are: 1) an unexpected exception occurs - close the relay ASAP. - Use excepthook 2) The interpreter gets sent a kill or similar unexpected termination - use atexit because finally may not get called. (I'm not sure, so belt n' braces here) (BTW Does anyone know what the interpreter does when suspending - Ctrl-Z in Unix land?) 3) Normal program exit. Use the finally clause. But its only ever going to be a best endeavour, that's why Python is not suitable for true real-time/critical apps... But I'd never trust any environment to its usual behaviour if there is a possibility of something being broken. In this case the relay and its battery pack. the program closes normally or the main code raises an exception, but not if the process is killed. What's not clear in the Python documentation is how Python responds to a kill(or suspend). I'd hope the atexit got called even in a kill. I would not expect the finally to be executed. Of course, if its a seg fault you are probably stuffed either way... I ran a few experiments: $ cat bnb.py import atexit import os import signal import sys import time def handle_except(*args): print(except, args, flush=True) def handle_exit(): print(exit, flush=True) def register_signalhandler(sig): def handler(*args): print(receiving signal, sig, args, flush=True) signal.signal(sig, handler) def main(): print(Hello from, os.getpid()) while True: print(., flush=True, end=) time.sleep(1) sys.excepthook = handle_except atexit.register(handle_exit) for sig in sys.argv[1:]: register_signalhandler(getattr(signal, sig)) try: main() finally: print(finally, flush=True) $ python3 bnb.py Hello from 32578 ^Cfinally except (class 'KeyboardInterrupt', KeyboardInterrupt(), traceback object at 0x7ff97b001bc8) exit When there is no signal handler all three mechanisms work, in the order - finally - except hook - exit handler Now let's kill: $ python3 bnb.py Hello from 32584 .Terminated None of the three are invoked. Let's install a signal handler for SIGTERM: $ python3 bnb.py SIGTERM Hello from 32593 .receiving signal 15 (15, frame object at 0x7f818e0bb648) ...^Cfinally except (class 'KeyboardInterrupt', KeyboardInterrupt(), traceback object at 0x7f818cdc6bc8) exit The signal is intercepted (and ignored by the no-op handler thus the additional Ctrl-C). If we raise a SystemExit in the handler - finally - exit handler will be invoked, but not the except hook. $ kill -9 of course cannot be intercepted. My conclusions: - If finally does not work nothing does. - Signal handlers increase safety Bonus: $ python3 bnb.py SIGTSTP Hello from 32614 ^Zreceiving signal 20 (20, frame object at 0x7f2f8a897648) ^Cfinally except (class 'KeyboardInterrupt', KeyboardInterrupt(), traceback object at 0x7f2f895a2bc8) exit So Ctrl-Z can be intercepted. The program could put the relay into a safe state before it suspends. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] UPDATE: Is there a 'hook' to capture all exits from a python program?
On 21Mar2015 09:19, Peter Otten __pete...@web.de wrote: I ran a few experiments: [...] Bonus: $ python3 bnb.py SIGTSTP Hello from 32614 ^Zreceiving signal 20 (20, frame object at 0x7f2f8a897648) ^Cfinally except (class 'KeyboardInterrupt', KeyboardInterrupt(), traceback object at 0x7f2f895a2bc8) exit So Ctrl-Z can be intercepted. The program could put the relay into a safe state before it suspends. Yes, Ctrl-Z (SIGTSTP) can be caught (stop from terminal, ^Z). Note, however, that SIGSTOP cannot be caught (stop); it is not SIGTSTP. Basicly you can kill (SIGKILL - abort process and never schedule it again), stop (SIGSTOP - cease scheduling this process) and continue (SIGCONT - resume scheduling this process) a process from outside a process and the process cannot intercept these. Which is just great! However, it means there are some things you cannot manage from within the process. This is where watchdogs of various kinds come into play: an external process of some kind which monitors the primary process (or something it manages), and take action if the primary process goes away or some activity does not occur for a period. Cheers, Cameron Simpson c...@zip.com.au Cordless hoses have been around for quite some time. They're called buckets. - Dan Prener pre...@watson.ibm.com ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] UPDATE: Is there a 'hook' to capture all exits from a python program?
Alan Gauld wrote: On 20/03/15 02:57, Doug Basberg wrote: Still, I would like to know if a 'hook' exists on exit from Python. I am running Linux on a Raspberry Pi with Python 2.7.4 I also run an Apache server on the Pi for monitor and control of power, HVAC, and security. Your previous mail got you three options. I'd use all of them! https://docs.python.org/3/library/atexit.html ... But that's only for normal program termination; sys.excepthook is for unexpected exits def close_relay(e=None,v=None,t=None): try: if not relay_closed() really_close_relay() except: really_close_relay() import sys, atexit atexit.register(close_relay) sys.excepthook = close_relay try: main program here finally: close_relay() That reeks of cargo cult. Are there actual scenarios for each of the three mechanisms where it is the only one that works? I would expect that try: main program here finally: close_relay() provides the same level of confidence, i. e. the relay will be closed when the program closes normally or the main code raises an exception, but not if the process is killed. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] UPDATE: Is there a 'hook' to capture all exits from a python program?
On Fri, Mar 20, 2015 at 08:35:34PM +, Alan Gauld wrote: Yeah, I know you can catch a signal and add your own handler, but I meant what is the default Python suspend behaviour? Does it execute any outstanding exception blocks? What about finally blocks? Or, if about to exit a context manager, the __exit__ method? Depends on the signal. I'm not an expert on how signals work in Linux, or other Unixes, but I expect that, in the absense of a specific signal handler to catch it, the sleep (pause?) signal will cause the interpreter to just stop and wait. I think that's signal 19 on Linux, and 18 to wake. Signal 9 doesn't give the interpreter to do anything. The OS just yanks the carpet out from under its feet and terminates the process with extreme prejudice. Signal 9 cannot be caught, no signal handlers will detect it, no try...finally blocks will run. The process just stops. Don't use kill -9 unless you need to. I always try three steps to kill a rogue process: First use kill processid with no other arguments. Give it 30 seconds or so to let the process tidy up after itself, and if it still hasn't quiet, try kill -HUP processid. Again, give it 30 seconds or so. Then, if and only if necessary, kill -9 processid. Or does it just stop and wait till its resumed? Kind of like an implicit yield statement? -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] UPDATE: Is there a 'hook' to capture all exits from a python program?
Hi, This is mostly a distant footnote to Doug Basberg's original question, which I believe is largely answered at this point. Albert-Jan Roskum, Alan Gauld and Steven D'Aprano were asking about signals and how they are handled (in Un*xen). I am trying to address that. Yeah, I know you can catch a signal and add your own handler, but I meant what is the default Python suspend behaviour? Does it execute any outstanding exception blocks? What about finally blocks? Or, if about to exit a context manager, the __exit__ method? Suspension of the process has nothing to do with Python. This happens in the Un*x scheduler--long before Python is involved. Depends on the signal. Correct. It all depends on the signal. Short version: (Apologies, Jack Nicholson, in demonic form or otherwise): * You can't 'handle': STOP, CONT, KILL, SEGV, BUS. * You can handle: HUP, INT, QUIT, USR1, USR2, PIPE, ALRM, TERM and others. Short advice (though I do not have experience dealing with signals in a realtime environment). I suggest the following guidelines: 1. Do not catch a signal you cannot handle (or do not intend to handle). 2. Do everything you can at startup to make sure that the environment in which you are operating is as you expect. 3. Catch all the signals you want to catch and, in response to receiving such a signal, do what you need in order to shut down cleanly. This coexists peacefully with the handy atexit handlers suggested earlier. (I am normally not at all a fan of an unspecific try--finally, but I get what Peter Otten is suggesting and might make the same choice, were I faced with Doug Basberg's situation.) I'm not an expert on how signals work in Linux, or other Unixes, but I expect that, in the absense of a specific signal handler to catch it, the sleep (pause?) signal will cause the interpreter to just stop and wait. I think that's signal 19 on Linux, and 18 to wake. Longer version: I have experience with handling Linux signals and Python. There may be subtle differences on other Un*xen. If you wish to know more, I would suggest reading the chapter on Signals in _Advanced Programming in the Unix Environment_ (Chapter 10, in my second edition by Stevens Rago). You cannot catch nor handle: * SIGSTOP (19), because that tells Un*x, Please remove this process from the scheduler, i.e. freeze it! * SIGCONT (18), because that tells Unix, Please restore this process to normal scheduling, i.e. unfreeze it. * SIGKILL (9), because that tells Unix, Terminate this thing, with prejudice! Do not tell it what happened. This means, your Un*X will never actually deliver SIGSTOP, SIGCONT or SIGKILL to Python and your program. I believe that you cannot do anything with the following signals: * SIGSEGV (11), because this means that there has been a memory fault. Python is a sufficiently high-level language, that, if this happens, this should not be your code doing it. (Unless you are writing C extensions for Python, and then, of course, you know what you are doing) * SIGBUS (7), because this is extraordinarily rare (today), but would be a case of trying to access memory that does not exist. In practice, I have seen SIGSEGV often over the last 20 years (perhaps I have worked with flaky software, or perhaps that is just something that happens in this line of work). I have seen SIGBUS very rarely (usually a precursor to a machine eating itself for lunch). The signals STOP and CONT are so rarely exhibited that they are perceived as exotic specimens when demonstrated. The KILL signal is the big hammer that everybody learns in their first month using any Un*x (which is unfortunate because of the power it commands). Signal 9 doesn't give the interpreter to do anything. The OS just yanks the carpet out from under its feet and terminates the process with extreme prejudice. Signal 9 cannot be caught, no signal handlers will detect it, no try...finally blocks will run. The process just stops. Correct. When the (Linux | Un*x) kernel has a signal 9 for a process, that process does not get any chance to clean up. It simply disappears. It is never given an opportunity to run again--i.e. it will never be scheduled again. Don't use kill -9 unless you need to. I always try three steps to kill a rogue process: First use kill processid with no other arguments. Give it 30 seconds or so to let the process tidy up after itself, and if it still hasn't quiet, try kill -HUP processid. Again, give it 30 seconds or so. Then, if and only if necessary, kill -9 processid. Agreed. In my experience, most mature sysadmins do this, too. Or does it just stop and wait till its resumed? Kind of like an implicit yield statement? Sending a SIGSTOP to a process is equivalent to freezing it in memory/process space. I sometimes think of this as suspend (because of ctrl-Z in bash,
Re: [Tutor] UPDATE: Is there a 'hook' to capture all exits from a python program?
On Fri, Mar 20, 2015 10:37 AM CET Peter Otten wrote: Alan Gauld wrote: On 20/03/15 02:57, Doug Basberg wrote: Still, I would like to know if a 'hook' exists on exit from Python. I am running Linux on a Raspberry Pi with Python 2.7.4 I also run an Apache server on the Pi for monitor and control of power, HVAC, and security. Your previous mail got you three options. I'd use all of them! https://docs.python.org/3/library/atexit.html ... But that's only for normal program termination; sys.excepthook is for unexpected exits def close_relay(e=None,v=None,t=None): try: if not relay_closed() really_close_relay() except: really_close_relay() import sys, atexit atexit.register(close_relay) sys.excepthook = close_relay try: main program here finally: close_relay() That reeks of cargo cult. Are there actual scenarios for each of the three mechanisms where it is the only one that works? I would expect that try: main program here finally: close_relay() Is this (also) called a diaper pattern? Or is that name reserved for the antipattern with try-bare except, where the 'except' catches all the sh*t (pardon my language)? provides the same level of confidence, i. e. the relay will be closed when the program closes normally or the main code raises an exception, but not if the process is killed. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] UPDATE: Is there a 'hook' to capture all exits from a python program?
Albert-Jan Roskam wrote: I would expect that try: main program here finally: close_relay() Is this (also) called a diaper pattern? Cool name, but most parents want the baby's faeces to go into the diapers whereas try ... bare except is notorious for swallowing useful information that should be propagated to the user or developer. Or is that name reserved for the antipattern with try-bare except, where the 'except' catches all the sh*t (pardon my language)? For try ... finally it's its raison d'être to execute some code no matter what. E. g. before the advent of with ... with open(...) as f: ... # use file # file is closed now even if something went wrong while using it and no # matter how garbage collection works for the Python implementation running # the code. # The exception (if any) is not swallowed it was good style to write f = open(...) try: ... # use file finally: f.close() # file is closed... ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] UPDATE: Is there a 'hook' to capture all exits from a python program?
On 20Mar2015 19:20, Martin A. Brown mar...@linux-ip.net wrote: [...] Short version: (Apologies, Jack Nicholson, in demonic form or otherwise): * You can't 'handle': STOP, CONT, KILL, SEGV, BUS. You can handle SEGV and BUS. Though probably not meaningfully in Python, haven't tried; if they fire in Python something internal is already badly wrong. IIRC, in the distant past the Bourne shell used SIGSEGV as a trigger to allocate more memory:-) Really, the only 3 a UNIX process can't intercept are the first three: STOP, CONT, KILL. Since everything else Martin says seems to be in the context of in Python, no other quibbles. Cheers, Cameron Simpson c...@zip.com.au ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] UPDATE: Is there a 'hook' to capture all exits from a python program?
On 20/03/15 09:37, Peter Otten wrote: def close_relay(e=None,v=None,t=None): try: if not relay_closed() really_close_relay() except: really_close_relay() The purpose of the if clause is to ensure that if the function is called many times you only close the relay once (I surmised that more than once could be harmful?) import sys, atexit atexit.register(close_relay) sys.excepthook = close_relay atexit should be overkill, but it might be needed if for some reason the interpreter dies while performing the usual cleanup. excepthook replaces the usual exception mechanism with a clean up. This is needed for cases where an exception occurs before getting to the finally. We want to close the relay ASAP, not waiting till the interpreter decides to call the finally. try: main program here finally: close_relay() This is the happy path where everything shuts down as expected. That reeks of cargo cult. Are there actual scenarios for each of the three mechanisms where it is the only one that works? In real-time you never trust anything. Always cover your back. I would expect that try: main program here finally: close_relay() provides the same level of confidence, Only if the interpreter is behaving as normal. The hooks are to try (we hope) to catch cases where the interpreter has broken its normal flow. So the scenarios are: 1) an unexpected exception occurs - close the relay ASAP. - Use excepthook 2) The interpreter gets sent a kill or similar unexpected termination - use atexit because finally may not get called. (I'm not sure, so belt n' braces here) (BTW Does anyone know what the interpreter does when suspending - Ctrl-Z in Unix land?) 3) Normal program exit. Use the finally clause. But its only ever going to be a best endeavour, that's why Python is not suitable for true real-time/critical apps... But I'd never trust any environment to its usual behaviour if there is a possibility of something being broken. In this case the relay and its battery pack. the program closes normally or the main code raises an exception, but not if the process is killed. What's not clear in the Python documentation is how Python responds to a kill(or suspend). I'd hope the atexit got called even in a kill. I would not expect the finally to be executed. Of course, if its a seg fault you are probably stuffed either way... -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] UPDATE: Is there a 'hook' to capture all exits from a python program?
On 20/03/15 20:04, Albert-Jan Roskam wrote: (BTW Does anyone know what the interpreter does when suspending - Ctrl-Z in Unix land?) No experience with it, but I would first check the 'signal' module Yeah, I know you can catch a signal and add your own handler, but I meant what is the default Python suspend behaviour? Does it execute any outstanding exception blocks? What about finally blocks? Or, if about to exit a context manager, the __exit__ method? Or does it just stop and wait till its resumed? Kind of like an implicit yield statement? -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] UPDATE: Is there a 'hook' to capture all exits from a python program?
- On Fri, Mar 20, 2015 8:05 PM CET Alan Gauld wrote: On 20/03/15 09:37, Peter Otten wrote: def close_relay(e=None,v=None,t=None): try: if not relay_closed() really_close_relay() except: really_close_relay() The purpose of the if clause is to ensure that if the function is called many times you only close the relay once (I surmised that more than once could be harmful?) import sys, atexit atexit.register(close_relay) sys.excepthook = close_relay atexit should be overkill, but it might be needed if for some reason the interpreter dies while performing the usual cleanup. excepthook replaces the usual exception mechanism with a clean up. This is needed for cases where an exception occurs before getting to the finally. We want to close the relay ASAP, not waiting till the interpreter decides to call the finally. try: main program here finally: close_relay() This is the happy path where everything shuts down as expected. That reeks of cargo cult. Are there actual scenarios for each of the three mechanisms where it is the only one that works? In real-time you never trust anything. Always cover your back. I would expect that try: main program here finally: close_relay() provides the same level of confidence, Only if the interpreter is behaving as normal. The hooks are to try (we hope) to catch cases where the interpreter has broken its normal flow. So the scenarios are: 1) an unexpected exception occurs - close the relay ASAP. - Use excepthook 2) The interpreter gets sent a kill or similar unexpected termination - use atexit because finally may not get called. (I'm not sure, so belt n' braces here) (BTW Does anyone know what the interpreter does when suspending - Ctrl-Z in Unix land?) No experience with it, but I would first check the 'signal' module import signal import sys def signal_term_handler(signal, frame): print 'got SIGTERM' sys.exit(0) signal.signal(signal.SIGTERM, signal_term_handler) https://nattster.wordpress.com/2013/06/05/catch-kill-signal-in-python/ 3) Normal program exit. Use the finally clause. But its only ever going to be a best endeavour, that's why Python is not suitable for true real-time/critical apps... But I'd never trust any environment to its usual behaviour if there is a possibility of something being broken. In this case the relay and its battery pack. the program closes normally or the main code raises an exception, but not if the process is killed. What's not clear in the Python documentation is how Python responds to a kill(or suspend). I'd hope the atexit got called even in a kill. I would not expect the finally to be executed. Of course, if its a seg fault you are probably stuffed either way... -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
