Re: [Python-Dev] Status of thread cancellation
Nick Maclaren [EMAIL PROTECTED] wrote: Well, I have seen it hundreds of times on a dozen different Unices; it is very common. You don't always SEE the stuck process - sometimes the 'kill -9' causes the pid to become invisible to ps etc., and just occasionally it can continue to use CPU until the system is rebooted. You're describing something caused by a buggy operating system. I have never seen any modern Unix exhibit any of the behaviours you describe. I have seen such things in the 1990's though. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of thread cancellation
Grrk. I have done this myself, and been involved in one of the VERY few commercial projects that attempted to do it properly (IBM CEL, the other recent one being VMS). I am afraid that there are a lot of misapprehensions here. Several people have said things like: The thing to model this on, I think, would be the BSD sigmask mechanism, which lets you selectively block certain signals to create a critical section of code. A context manager could be used to make its use easier and less error-prone (i.e. harder to block async exceptions and then forget to unblock them). No, no, no! That is an TRULY horrible! It works fairly well for things like device drivers, which are both structurally simple and with no higher level recovery mechanism, so that a failure turning into a hard hang is not catastrophic. But it is precisely what you DON'T want for complex applications, especially when a thread may need to call an external service 'non-interruptibly'. Think of updating a complex object in a multi-file database, for example. Interrupting half-way through leaves the database in a mess, but blocking interrupts while (possibly remote) file updates complete is asking for a hang. You also see it in horrible GUI (including raw mode text) programs that won't accept interrupts until you have completed the action they think you have started. One of the major advantages of networked systems is that you can usually log in remotely and kill -9 the damn process! The way that I, IBM and DEC approached it was by the classic callback mechanism, with a carefully designed way of promoting unhandled exceptions/interrupts. For example, the following is roughly what I did (somewhat extended, as I didn't do all of this for all exceptions): An event set a defined flag, which could be tested (and cleared) by the thread. If a second, similar event arrived (or it was not handled after a suitable time), the event was escalated. If so, a handler was called that HAD to return (again within a specific time). If a second, similar event arrived or it didn't return by a suitable time, the event was escalated. If so, another handler was called that COULDN'T return. If another event arrived, it returned, or it failed to close down the thread, the event was escalated. If so, the thread's built-in environment was closed down without giving the thread a chance to intervene. If that failed, the event was escalated. If so, the thread was frozen and process termination started. If clean termination failed, the event was escalated. If so, the run-time system produced a dump and killed itself. You can implement a BSD-style ignore by having an initial handler that just clears the flag and returns, but a third interrupt before it does so will force close-down. There was also a facility to escalate an exception at the point of generation, which could be useful for forcible closedown. There are a zillion variations of the above, but all mainframe experience is that callbacks are the only sane way to approach the problem IN APPLICATIONS. In kernel code, that is not so, which is why so many of the computer scientists design BSD-style handling (i.e. they think of kernel programming rather than very complex application programming). Unconditionally killing a whole process is no big problem because all the resources it's using get cleaned up by the OS, and the effect on other processes is minimal and well-defined (pipes and sockets get EOF, etc.). But killing a thread can leave the rest of the program in an awkward state. I wish that were so :-( Sockets, terminals etc. are stateful devices, and killing a process can leave them in a very unclean state. It is one of the most common causes of unkillable processes (the process can't go until its files do, and the socket is jammed). Many people can witness the horrible effects of ptys being left in 'echo off' or worse states, the X focus being left in a stuck override redirect window and so on. But you also have the multi-file database problem, which also applies to shared memory segments. Even if the process dies cleanly, it may be part of an application whose state is global across many processes. One common example is adding or deleting a user, where an unclean kill can leave the system in a very weird state. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: [EMAIL PROTECTED] Tel.: +44 1223 334761Fax: +44 1223 334679 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of thread cancellation
Nick Maclaren [EMAIL PROTECTED] wrote: Sockets, terminals etc. are stateful devices, and killing a process can leave them in a very unclean state. It is one of the most common causes of unkillable processes (the process can't go until its files do, and the socket is jammed). Can you elaborate on this? You can get zombie entries in the process table if nobody's called 'wait()' on them, and you can (extremely rarely) get unkillable process in 'disk-wait' state (usually due to hardware failure or a kernel bug, I suspect), but I've never heard of a process on a Unix-like system being unkillable due to something to do with sockets (or any other kind of file descriptor for that matter). How could a socket be 'jammed'? What does that even mean? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of thread cancellation
Jon Ribbens [EMAIL PROTECTED] wrote: Can you elaborate on this? You can get zombie entries in the process table if nobody's called 'wait()' on them, and you can (extremely rarely) get unkillable process in 'disk-wait' state (usually due to hardware failure or a kernel bug, I suspect), but I've never heard of a process on a Unix-like system being unkillable due to something to do with sockets (or any other kind of file descriptor for that matter). How could a socket be 'jammed'? What does that even mean? Well, I have seen it hundreds of times on a dozen different Unices; it is very common. You don't always SEE the stuck process - sometimes the 'kill -9' causes the pid to become invisible to ps etc., and just occasionally it can continue to use CPU until the system is rebooted. That is rare, however, and it normally just hangs onto locks, memory and other such resources. Very often its vampiric status is visible only because such things haven't been freed, or when you poke through kernel structures. Sockets get jammed because they are used to connect to subprocesses or kernel threads, which in turn access unreliable I/O devices. If there is a glitch on the device, the error recovery very often fails to work, cleanly, and may wait for an event that will never occur or go into a loop (usually a sleep/poll loop). Typically, a HIGHER level then times out the failing error recovery, so that the normal programmer doesn't notice. But it very often fails to kill the lower level code. As far as applications are concerned, a jammed socket is one where the higher level recovery has NOT done that, and is waiting for the lower level to complete - which it isn't going to do! The other effect that ordinary programmers notice is a system very gradually starting to run down after days/weeks/months of continual operation. The state is cleared by rebooting. Regards, Nick Maclaren. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of thread cancellation
Nick Maclaren wrote: Think of updating a complex object in a multi-file database, for example. Interrupting half-way through leaves the database in a mess, but blocking interrupts while (possibly remote) file updates complete is asking for a hang. Currently, threads can't be interrupted at all, so by this argument, the status quo is that we're always asking for a hang. My (possibly naive) thought would be with interrupts disabled: begin transaction try: with interrupts enabled: do the transaction except Interrupt: roll back the transaction raise else: commit the transaction Generally, I would approach things by having interrupts disabled in the top layers of the thread, and enabling them explicitly in regions where I'm prepared to handle them. So I'd be happy if they were disabled by default in a new thread. I'm not convinced by the argument that kernel programming is somehow different from application programming in this area. Seems to me they're both dealing with the same problem -- concurrent interacting processes and trying to make sure nothing can get irretrievably jammed up. So I can't see why similar techiques can't be used to solve the problems. Sockets, terminals etc. are stateful devices, and killing a process can leave them in a very unclean state. I agree that ttys are far too stateful -- if I were designing an OS I'd do them differently (more like STREAMS with pluggable layers). But I've never noticed any problem with statefulness of sockets. Maybe I don't use them in a fancy enough way. the X focus being left in a stuck override redirect window and so on. That's a misfeature of X, IMO -- it shouldn't be possible for a client to grab the entire display, only the windows that it owns. And there should always be a way of forcibly killing a client using nothing but the X server itself. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of thread cancellation
Nick Maclaren wrote: You don't always SEE the stuck process - sometimes the 'kill -9' causes the pid to become invisible to ps etc., and just occasionally it can continue to use CPU until the system is rebooted. If that happens, there's a bug in the kernel. A process killed with -9 shouldn't be using *any* resources at all, other than a tiny piece of kernel memory for its process structure until it gets reaped. Sockets get jammed because they are used to connect to subprocesses or kernel threads, which in turn access unreliable I/O devices. But it's not the *socket* which is jammed -- if you kill the process on the other end of it, anything connected to the socket will get EOF or SIGPIPE. (If you can't kill the process on the other end, even with -9, then again you have a kernel bug.) -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of thread cancellation
Josiah Carlson wrote: Greg Ewing [EMAIL PROTECTED] wrote: [EMAIL PROTECTED] wrote: Can you suggest any use-cases for thread termination which will *not* result in a completely broken and unpredictable heap after the thread has died? Suppose you have a GUI and you want to launch a long-running computation without blocking the user interface. You don't know how long it will take, so you want the user to be able to cancel it if he gets bored. If the code is in Python, you can use sys.settrace to handle this. If the code is in an extension module that a user has control over, having a cancel_thread() function that is made available to Python, and having your C code check the value of a single variable every few seconds could do the same thing (even checking the value in a tight loop shouldn't slow computations down significantly, branch prediction should be able to make it a more or less zero-cost operation). Yes, it can be tedious, but at least the programmer can actually control cleanup in a reasonable manner. Option 3, farm the long running operation out to another process and use the OS-provided facilities to abort and cleanup if the user changes their mind. It's the only way to be sure the aborted operation doesn't leave the main process in a dodgy state. Cheers, Nick. -- Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia --- http://www.boredomandlaziness.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of thread cancellation
On 16/03/2007 1.06, Greg Ewing wrote: Can you suggest any use-cases for thread termination which will *not* result in a completely broken and unpredictable heap after the thread has died? Suppose you have a GUI and you want to launch a long-running computation without blocking the user interface. You don't know how long it will take, so you want the user to be able to cancel it if he gets bored. There's no single place in the code where you could put in a check for cancellation. Sprinkling such checks all over the place would be tedious, or even impossible if large amounts of time are spent in calls to a third-party library that wasn't designed for such things. Interaction with the rest of the program is extremely limited -- some data is passed in, it churns away, and some data is returned. It doesn't matter what happens to its internal state if it gets interrupted, as it's all going to be thrown away. In that situation, it doesn't seem unreasonable to me to want to be able to just kill the thread. I don't see how it could do any more harm than using KeyboardInterrupt to kill a program, because that's all it is -- a subprogram running inside your main program. How would you handle this situation? It's really simple: don't use threads, use processes! Spawn an external process which does the calculation, pass data to it through pipe/socket/namedpipe/xmlrpc/whatever and read data back from it when it's done. If you need to kill it, just kill it away, at any asynchronous time: the OS will clean up after it. After many years working with these issues, I came to the personal conclusion of avoiding threads as much as possible. Threads are processes with shared memory, but in many real-world use cases I faced, there is really only a very little chunk of memory which is shared, and Python makes it incredibly easy to marshal data to a process (pickle or whatever). So in many cases there's really little excuses for going mad with threads. -- Giovanni Bajo ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of thread cancellation
On 12:06 am, [EMAIL PROTECTED] wrote: [EMAIL PROTECTED] wrote: Can you suggest any use-cases for thread termination which will *not* result in a completely broken and unpredictable heap after the thread has died? Suppose you have a GUI and you want to launch a long-running computation without blocking the user interface. You don't know how long it will take, so you want the user to be able to cancel it if he gets bored. That's a perfectly reasonable use-case which doesn't require this feature at all ;). Interaction with the rest of the program is extremely limited -- some data is passed in, it churns away, and some data is returned. It doesn't matter what happens to its internal state if it gets interrupted, as it's all going to be thrown away. If that's true, then the state-sharing features of threads aren't required, which is the right way to design concurrent software anyway. In that situation, it doesn't seem unreasonable to me to want to be able to just kill the thread. I don't see how it could do any more harm than using KeyboardInterrupt to kill a program, because that's all it is -- a subprogram running inside your main program. The key distinction between threads and processes is the sharing of internal program state. How would you handle this situation? Spawn a process, deliver the result via an event. If you're allergic to event-driven programming, then you can spawn a process *in* a thread, and block in the thread on reading from the process's output, then kill the *process* and have that terminate the output, which terminates the read(). This is a lot like having a queue that you can put a stop object into, except the file interface provided by OSes is kind of crude. Still no need to kill the thread. At the end of the day though, you're writing a GUI in this use-case and so you typically *must* be cognizant of event-driven issues anyway. Many GUIs (even in the thread-happy world of Windows) aren't thread-safe except for a few specific data-exchange methods, which behave more or less like a queue. One of the 35 different existing ways in which one can spawn a process from Python, I hope, will be sufficient for this case :). ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Status of thread cancellation
I just proposed to implement thread cancellation for the SoC. Is there any prior work where one could start? Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of thread cancellation
On Thu, 15 Mar 2007 14:34:15 +0100, \Martin v. Löwis\ [EMAIL PROTECTED] wrote: I just proposed to implement thread cancellation for the SoC. Is there any prior work where one could start? The outcome of some prior work, at least: http://java.sun.com/j2se/1.4.2/docs/guide/misc/threadPrimitiveDeprecation.html Jean-Paul ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of thread cancellation
I just proposed to implement thread cancellation for the SoC. Is there any prior work where one could start? Jean-Paul The outcome of some prior work, at least: Jean-Paul http://java.sun.com/j2se/1.4.2/docs/guide/misc/threadPrimitiveDeprecation.html I responded to that. I got the impression reading that page that the killed thread doesn't regain control so it can't clean up its potentially inconsistent data structures. I inferred from Martin's proposal that he expected the thread to be able to catch the exception. Perhaps he can elaborate on what cleanup actions the dying thread will be allowed to perform. Skip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of thread cancellation
On Thu, 15 Mar 2007 09:41:31 -0500, [EMAIL PROTECTED] wrote: I just proposed to implement thread cancellation for the SoC. Is there any prior work where one could start? Jean-Paul The outcome of some prior work, at least: Jean-Paul http://java.sun.com/j2se/1.4.2/docs/guide/misc/threadPrimitiveDeprecation.html I responded to that. I got the impression reading that page that the killed thread doesn't regain control so it can't clean up its potentially inconsistent data structures. The second question on the page: Couldn't I just catch the ThreadDeath exception and fix the damaged object? Addresses this. I inferred from Martin's proposal that he expected the thread to be able to catch the exception. Perhaps he can elaborate on what cleanup actions the dying thread will be allowed to perform. Perhaps he can. Hopefully, he can specifically address these points: 1. A thread can throw a ThreadDeath exception almost anywhere. All synchronized methods and blocks would have to be studied in great detail, with this in mind. 2. A thread can throw a second ThreadDeath exception while cleaning up from the first (in the catch or finally clause). Cleanup would have to repeated till it succeeded. The code to ensure this would be quite complex. Jean-Paul ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of thread cancellation
Jean-Paul Calderone schrieb: I inferred from Martin's proposal that he expected the thread to be able to catch the exception. Perhaps he can elaborate on what cleanup actions the dying thread will be allowed to perform. Perhaps he can. Hopefully, he can specifically address these points: 1. A thread can throw a ThreadDeath exception almost anywhere. All synchronized methods and blocks would have to be studied in great detail, with this in mind. 2. A thread can throw a second ThreadDeath exception while cleaning up from the first (in the catch or finally clause). Cleanup would have to repeated till it succeeded. The code to ensure this would be quite complex. Clearly, a thread need to have its finally blocks performed in response to a cancellation request. These issues are real, however, they apply to any asynchronous exception, not just to thread cancellation. In Python, we already have an asynchronous exception: KeyboardInterrupt. This suffers from the same problems: a KeyboadInterrupt also can occur at any point, interrupting code in the middle of its finally-blocks. The other exception that is nearly-asynchronous is OutOfMemoryError, which can occur at nearly any point (but of course, never occurs in practice). So yes, it would be good if Python's exception handling supported asynchronous exceptions in a sensible way. I have to research somewhat more, but I think the standard solution to the problem in operating system (i.e. disabling interrupts at certain points, explicitly due to code or implicitly as a result of entering the interrupt handler) may apply. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of thread cancellation
Martin v. Löwis wrote: asynchronous exceptions in a sensible way. I have to research somewhat more, but I think the standard solution to the problem in operating system (i.e. disabling interrupts at certain points, explicitly due to code or implicitly as a result of entering the interrupt handler) may apply. Two already working schemes, that are similar, comes to my mind. One is signals in Linux/Unix, where you can send SIGTERM, and the process can handle it and do whatever it takes. But also you can send SIGKILL, which can not be blocked. The other is microprocessors, where you have interrupts, and when the interrupt is received, you disable it (are there processors that support reentrant interrupts? I don't know of any, but I'm no specialist here). To me, is natural this behaviour: One can send ThreadDeath to the thread, and it can handle it or no. If not, it dies. If yes, it does some stuff and dies. But if I send a second ThreadDeath to the same thread, when it's still dying, for me it's ok to receive an answer like Ok, ok, I heard you, I'm on it. But, in that scenario, should be a way to say to the thread Die, die now, no matter what? Regards, -- . Facundo . Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of thread cancellation
On 04:24 pm, [EMAIL PROTECTED] wrote:
Jean-Paul Calderone schrieb:
I inferred from Martin's proposal that he
expected the thread to be able to catch the exception. Perhaps he
can
elaborate on what cleanup actions the dying thread will be allowed to
perform.
Perhaps he can. Hopefully, he can specifically address these points:
1. A thread can throw a ThreadDeath exception almost anywhere. All
synchronized methods and blocks would have to be studied in
great
detail, with this in mind.
2. A thread can throw a second ThreadDeath exception while cleaning
up
from the first (in the catch or finally clause). Cleanup would
have
to repeated till it succeeded. The code to ensure this would be
quite
complex.
Clearly, a thread need to have its finally blocks performed in response
to a cancellation request. These issues are real, however, they apply
to any asynchronous exception, not just to thread cancellation.
To be sure, the problem does apply to all asynchronous exceptions.
That's why it is generally understood that a program which has received
an asynchronous exception cannot continue.
In Python, we already have an asynchronous exception:
KeyboardInterrupt.
This suffers from the same problems: a KeyboadInterrupt also can occur
at any point, interrupting code in the middle of its finally-blocks.
The other exception that is nearly-asynchronous is OutOfMemoryError,
which can occur at nearly any point (but of course, never occurs in
practice).
KeyboardInterrupt and MemoryError share a common feature which forced
thread termination does not: nobody reasonably expects the program to
keep running after they have been raised. Indeed, programs are written
with the expectation that MemoryError will never occur, and if it does,
the user is not surprised to find them in an inconsistent state. In any
situation where a MemoryError may reasonably be expected - that is to
say, a specific, large allocation of a single block of memory - it can
be trapped as if it were not asynchronous.
Long-running Python programs which expect to need to do serious clean-up
in the face of interrupts, in fact, block KeyboardInterrupt by
registering their own interrupt handlers (Zope, Twisted).
Developers who think they want thread cancellation inevitably believe
they can, if they are sufficiently careful, implement operating-
system-like scheduling features by starting arbitrary user code and then
providing terminate, pause, and resume commands. That was the
original intent of these (now removed) Java APIs, and that is why they
were removed: you can't do this. It's impossible.
Asynchronous exceptions are better than immediate termination because
they allow for code which is allocating scarce or fragile resources to
have a probabilistically better chance of cleaning up. However, nobody
writes code like this:
def addSomeStuff(self, volume, mass):
self.volume += volume
try:
self.mass += mass
except AsynchronousInterrupt:
while 1:
try:
self.volume -= volume
break
except AsynchronousInterrupt:
pass
and nobody is going to start if the language provides thread
termination. Async-Exception-Safe Python code is, and will be, as rare
as POSIX Async-Safe functions, which means at best you will be able to
call a thread cancellation API and have it _appear_ to work in some
circumstances. In any system which uses Python code not explicitly
designed to support asynchronous exceptions (let's say, the standard
library) it will be completely impossible to write correct code.
I'm not a big fan of shared-state-threading, but it does allow for a
particular programming model. Threading provides you some guarantees.
You can't poke around on the heap, but you know that your stack, and
your program counter, are inviolate. You can reason about, if not quite
test, the impact of sharing a piece of state on the heap; its
destructive methods need to be synchronized along with the read methods
that interact with it.
Asynchronous exceptions destroy all hope of sanity. Your program might
suddenly perform a nonlocal exit _anywhere_ except, maybe, inside a
finally block. This literally encourages some people that program in
environments where asynchronous exceptions are possible (.NET, in
particular) to put huge chunks of application code inside finally
blocks. They generally look like this:
try {}
finally {
// entire application here
}
because that is really the only way you can hope to write code that will
function robustly in such an environment.
So yes, it would be good if Python's exception handling supported
asynchronous exceptions in a sensible way. I have to research somewhat
more, but I think the standard solution to the problem in operating
system (i.e. disabling interrupts at certain points, explicitly
due to code or implicitly as a result of
Re: [Python-Dev] Status of thread cancellation
[EMAIL PROTECTED] schrieb: Just in case it's not clear from the other things I've said: this is a terrible, terrible idea, and I am shocked that it is even being *considered* for inclusion in Python. As a foolish youth, I wasted many months trying to get a program that used Java's (then not deprecated) asynchronous exception APIs to behave properly. It wasn't possible then, and it isn't possible now. Ok, I withdraw this SoC project idea. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of thread cancellation
Facundo Batista wrote: are there processors that support reentrant interrupts? The PDP11 had seven priority levels for interrupts. When an interrupt was handled, interrupts with priorities less than or equal to the current level were blocked, but the handler could be interrupted by a higher priority interrupt. Also, on any processor I know about, there's nothing to stop an interrupt handler re-enabling interrupts once it's ensured that the particular one it's handling isn't going to happen again. You can use this to implement an interrupt priority scheme in software if the hardware doesn't support it. So yes, re-entrant interrupts do make sense in some situations. The thing to model this on, I think, would be the BSD sigmask mechanism, which lets you selectively block certain signals to create a critical section of code. A context manager could be used to make its use easier and less error-prone (i.e. harder to block async exceptions and then forget to unblock them). But, in that scenario, should be a way to say to the thread Die, die now, no matter what? Unconditionally killing a whole process is no big problem because all the resources it's using get cleaned up by the OS, and the effect on other processes is minimal and well-defined (pipes and sockets get EOF, etc.). But killing a thread can leave the rest of the program in an awkward state. I'm inclined to think that there should be some way to do it, and any locks held by the killed thread should be broken. It's then up to the program to deal with the consequences. If it's not willing to do that, then it shouldn't use the instant-death mechanism. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of thread cancellation
[EMAIL PROTECTED] wrote: Can you suggest any use-cases for thread termination which will *not* result in a completely broken and unpredictable heap after the thread has died? Suppose you have a GUI and you want to launch a long-running computation without blocking the user interface. You don't know how long it will take, so you want the user to be able to cancel it if he gets bored. There's no single place in the code where you could put in a check for cancellation. Sprinkling such checks all over the place would be tedious, or even impossible if large amounts of time are spent in calls to a third-party library that wasn't designed for such things. Interaction with the rest of the program is extremely limited -- some data is passed in, it churns away, and some data is returned. It doesn't matter what happens to its internal state if it gets interrupted, as it's all going to be thrown away. In that situation, it doesn't seem unreasonable to me to want to be able to just kill the thread. I don't see how it could do any more harm than using KeyboardInterrupt to kill a program, because that's all it is -- a subprogram running inside your main program. How would you handle this situation? If you can think of such a case, are you sure it wouldn't be better served by a set of threads communicating over queues and sending 'Stop' objects to each other If the thread is guaranteed to return to reading from the queue within a bounded time, that's fine, and it's the solution I would recommend in that case. But not all cases are like that. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of thread cancellation
Greg Ewing [EMAIL PROTECTED] wrote: [EMAIL PROTECTED] wrote: Can you suggest any use-cases for thread termination which will *not* result in a completely broken and unpredictable heap after the thread has died? Suppose you have a GUI and you want to launch a long-running computation without blocking the user interface. You don't know how long it will take, so you want the user to be able to cancel it if he gets bored. If the code is in Python, you can use sys.settrace to handle this. If the code is in an extension module that a user has control over, having a cancel_thread() function that is made available to Python, and having your C code check the value of a single variable every few seconds could do the same thing (even checking the value in a tight loop shouldn't slow computations down significantly, branch prediction should be able to make it a more or less zero-cost operation). Yes, it can be tedious, but at least the programmer can actually control cleanup in a reasonable manner. The only case that I have been able to come up with that is not covered with these two is if you have no control over the C-level code, which would be the case in a precompiled 3rd party extension or system call. In the system call case, I'm not sure there is a sane way to abort it on all platforms, and I can just about guarantee that even if you *could* kill a thread, doing so in 3rd party code (depending on the code) could leave you in a questionable state (memory leaks, temporary files, broken data structures, etc.). It seems better to write to allow for cancellation, rather than adding a big red STOP button to threads. - Josiah ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
