Re: Thread._stop() behavior changed in Python 3.4
On Mar 17, 2014 12:53 PM, "Jurko Gospodnetić" wrote: > > Hi. > > > On 17.3.2014. 19:03, Ian Kelly wrote: >> >> So yes, despite the lack of true concurrency, a thread can continue to >> run after its _stop has been called. > > > Actually 'true' or 'false' concurrency does not matter here. > > CPython supports multiple threads and implements them using underlying native OS threads. The fact that it has an internal mutex (GIL) preventing it from executing/interpreting Python code at the same time in multiple threads (most of the time) does not come into play.. When one thread exits its GIL protected section (e.g. finishes processing one bytecode instruction and is about to go on to processing the next one), another thread may pick up the GIL and do some of its work, e.g. print out some output. Yes, and whenever a thread acquires the GIL it *could* check whether its _stop flag has been set before it starts executing any Python code. Apparently though it does not, perhaps for performance reasons. -- https://mail.python.org/mailman/listinfo/python-list
Re: Thread._stop() behavior changed in Python 3.4
Hi. On 17.3.2014. 19:03, Ian Kelly wrote: So yes, despite the lack of true concurrency, a thread can continue to run after its _stop has been called. Actually 'true' or 'false' concurrency does not matter here. CPython supports multiple threads and implements them using underlying native OS threads. The fact that it has an internal mutex (GIL) preventing it from executing/interpreting Python code at the same time in multiple threads (most of the time) does not come into play.. When one thread exits its GIL protected section (e.g. finishes processing one bytecode instruction and is about to go on to processing the next one), another thread may pick up the GIL and do some of its work, e.g. print out some output. Best regards, Jurko Gospodnetić -- https://mail.python.org/mailman/listinfo/python-list
Re: Thread._stop() behavior changed in Python 3.4
On Tue, Mar 18, 2014 at 5:13 AM, Felix Yan wrote: > On Tuesday, March 18, 2014 05:08:20 Chris Angelico wrote: >> I've posted comments on both the issues you linked to. My guess based >> on a cursory look at paramiko is that it's a test suite watchdog, >> which would be much better implemented with a subprocess; I may be >> wrong, though. In any case, if it's just a tests problem, you should >> theoretically be able to ignore it. >> >> ChrisA > > I was just trying to comment and see yours... Thanks a lot! :D Your comment will mean more, since you actually use the thing :) ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Thread._stop() behavior changed in Python 3.4
On Tuesday, March 18, 2014 05:08:20 Chris Angelico wrote: > I've posted comments on both the issues you linked to. My guess based > on a cursory look at paramiko is that it's a test suite watchdog, > which would be much better implemented with a subprocess; I may be > wrong, though. In any case, if it's just a tests problem, you should > theoretically be able to ignore it. > > ChrisA I was just trying to comment and see yours... Thanks a lot! :D Regards, Felix Yan signature.asc Description: This is a digitally signed message part. -- https://mail.python.org/mailman/listinfo/python-list
Re: Thread._stop() behavior changed in Python 3.4
On Tue, Mar 18, 2014 at 4:59 AM, Felix Yan wrote: > For now I just skipped the test suites for paramiko to get the packaging done > (since the test suites themselves are passed without a problem, just the test > script made something wrong). I'll try to follow up the issue for paramiko :) I've posted comments on both the issues you linked to. My guess based on a cursory look at paramiko is that it's a test suite watchdog, which would be much better implemented with a subprocess; I may be wrong, though. In any case, if it's just a tests problem, you should theoretically be able to ignore it. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Thread._stop() behavior changed in Python 3.4
On Mon, Mar 17, 2014 at 11:40 AM, Chris Angelico wrote:
> Antoine says that this doesn't even stop the thread
> (I can't say; I've never used _stop(), for obvious reasons), so this
> code was doubly broken.
I was curious about that -- after all, Python's threads aren't truly
concurrent, so perhaps they could just test the flag each time they
resume -- so I tested it using 3.3. First I tried simply adding a
print call on to the end of the OP's function:
>>> def stale():
... import time
... time.sleep(1000)
... print('hello')
...
>>> t = threading.Thread(target=stale)
>>> t.start(); t._stop()
No output was printed, so at least a sleeping thread can apparently be
stopped. Then I tried removing the sleep call:
>>> def stale():
... for i in range(10): print('hello')
...
>>> t = threading.Thread(target=stale)
>>> t.start(); print('Starting'); t._stop(); print('Stopping')
hello
Starting
Stopping
>>> hello
hello
hello
hello
hello
hello
hello
hello
hello
So yes, despite the lack of true concurrency, a thread can continue to
run after its _stop has been called.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Thread._stop() behavior changed in Python 3.4
On Monday, March 17, 2014 17:33:09 Antoine Pitrou wrote: > Hi, > > Felix Yan gmail.com> writes: > > A minimized snippet to reproduce: > > > > #!/usr/bin/python > > import threading > > > > def stale(): > > import time > > time.sleep(1000) > > > > t = threading.Thread(target=stale) > > t.start() > > t._stop() > > > > This works correctly with Python 3.3, the program exits immediately after > > t._stop() called, and no exception was raised. > > Basically what you are doing is abusing a private method because you want > to make the thread daemonic after it was started (a daemonic thread is > not waited for at interpreter exit). Please do note one thing: the _stop() > method does *not* actually stop the thread; it just marks it stopped, but > the underlying OS thread continues to run (and may indeed continue to > execute Python code until the interpreter exits). > > So the obvious "solution" here is to mark the thread daemonic before > starting it. > > A possible related improvement would be to relax the contraints on > Thread.daemon to allow setting the flag on a running thread? > > That said, daemon threads (or abuse of the _stop() method as you did) can > lead to instabilities and oddities as some code will continue executing > while the interpreter starts shutting down. This has been improved but > perhaps not totally solved in recent interpreter versions. A fully correct > solution would involve gracefully telling the thread to shut down, via a > boolean flag, an Event, a file descriptor or any other means. > > (if you are interested in this, please open a new issue at > http://bugs.python.org) > > Regards > > Antoine. Thanks for the detailed explanation! Actually I didn't used _stop() myself either, but noticed the problem when trying to build paramiko against python 3.4. Thanks especially for the tip that the threads may be still running - actually I didn't even think about this part! For now I just skipped the test suites for paramiko to get the packaging done (since the test suites themselves are passed without a problem, just the test script made something wrong). I'll try to follow up the issue for paramiko :) Regards, Felix Yan signature.asc Description: This is a digitally signed message part. -- https://mail.python.org/mailman/listinfo/python-list
Re: Thread._stop() behavior changed in Python 3.4
On Tue, Mar 18, 2014 at 4:18 AM, Felix Yan wrote: > I noticed a behavior change on Thread._stop() with Python 3.4. > > I know the method is an undocumented "feature" itself, but some projects are > using it, and now they fail. > > I know trying to forcefully stop a thread is not really a good practice, but I > still wonder if there's an easy way to get broken programs to work again, just > in the way they currently are? You're using something that has a leading underscore on the name. Frankly, you shouldn't be doing that. Your code was already broken, before 3.4 came along, and it's just that 3.4 highlighted that brokenness. The PyCharm report that this is a Python 3.4 bug is simply incorrect. This code should be a strong indication that something's reaching into places it shouldn't be: https://github.com/JetBrains/intellij-community/blob/master/python/helpers/pydev/pydevd_comm.py#L109 The solution is to have the thread acknowledge, in some way, that it needs to shut down. Forcefully stopping a thread is actually a really bad practice, at least in Python (with OS/2 and VX-REXX, it's a different matter). Antoine says that this doesn't even stop the thread (I can't say; I've never used _stop(), for obvious reasons), so this code was doubly broken. The change in 3.4 should be an excuse to fix the code so it actually works, and works according to the spec, which will mean it works on all versions. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Thread._stop() behavior changed in Python 3.4
Hi, Felix Yan gmail.com> writes: > > A minimized snippet to reproduce: > > #!/usr/bin/python > import threading > def stale(): > import time > time.sleep(1000) > t = threading.Thread(target=stale) > t.start() > t._stop() > > This works correctly with Python 3.3, the program exits immediately after > t._stop() called, and no exception was raised. Basically what you are doing is abusing a private method because you want to make the thread daemonic after it was started (a daemonic thread is not waited for at interpreter exit). Please do note one thing: the _stop() method does *not* actually stop the thread; it just marks it stopped, but the underlying OS thread continues to run (and may indeed continue to execute Python code until the interpreter exits). So the obvious "solution" here is to mark the thread daemonic before starting it. A possible related improvement would be to relax the contraints on Thread.daemon to allow setting the flag on a running thread? That said, daemon threads (or abuse of the _stop() method as you did) can lead to instabilities and oddities as some code will continue executing while the interpreter starts shutting down. This has been improved but perhaps not totally solved in recent interpreter versions. A fully correct solution would involve gracefully telling the thread to shut down, via a boolean flag, an Event, a file descriptor or any other means. (if you are interested in this, please open a new issue at http://bugs.python.org) Regards Antoine. -- https://mail.python.org/mailman/listinfo/python-list
