[Peter Otten] > What I believe to be a minimal example: > > <freeze.py> > import Queue > import threading > import time > > q = Queue.Queue(4) > > def proc(): > while True: > q.get(1) > Queue.Queue() > print "YADDA" > > threading.Thread(target=proc).start() > > while True: > print "yadda" > q.put(None) > time.sleep(1) > </freeze.py> > > <freezemain.py> > import freeze > </freezemain.py>
CPython has an internal, reentrant import lock. When a thread does an import, it acquires this lock, and the lock remains held until that import is complete. As a consequence, no *other* thread can do an import (it blocks waiting to obtain the internal import lock) until the original import completes. So until "import freeze" returns in the main thread, no other thread can do an import. Partly for that reason, it's generally a Horrible Idea to start a thread as a side effect of importing a module. That's what freeze.py does, and you get the expected deadlock as a result. The main thread is hung waiting for "import freeze" to return, and the spawned thread is hung at an import in Queue.__init__() waiting for the main thread to release the import lock. > Invoking freezemain.py produces > > yadda > yadda > yadda > yadda > yadda > yadda > > i. e. consistently q.maxsize + 2. One item about to be put, one > already taken before Queue.Queue(). Deferring execution of the > module-level code until after the import > > <nofreeze.py> > import Queue > import threading > import time > > q = Queue.Queue(4) > > def proc(): > while True: > q.get(1) > Queue.Queue() > print "YADDA" > > def start(): > threading.Thread(target=proc).start() > > while True: > print "yadda" > q.put(None) > time.sleep(1) > </nofreeze.py> > > <nofreezemain.py> > import nofreeze > nofreeze.start() > </nofreezemain.py> > > and invoking nofreezemain.py produces > yadda > YADDA > yadda > YADDA > yadda > YADDA > yadda > YADDA > > apparently ad infinitum. Right. Note that this is the same reason threaded tests in Python's standard regression suite define a 'test_main()' function, called by the regrtest.py driver after import of the test's module completes. It's generally suicidal to start a thread as a side effect of an import. ... > Import _is_ a sensitive phase... It's quite easy to avoid thread problems in imports: never start a thread as a side effect of importing, and you'll never get a deadlock due to importing. > As an experiment I moved > > try: > import thread > except ImportError: > import dummy_thread as thread > > from the Queue.Queue.__init__() method to the module body -- > and now freezemain.py seems to work, too. So that would be an > easy remedy, but sure there is a reason why that import > statement is in such an unusual place? I think you'd have to ask Brett (who did most of the work on dummy_thread and dummy_threading). It doesn't really matter, though: it's a general truth that starting a thread as a side effect of importing is a recipe for deadlock, and hacking specific methods and functions to avoid imports just moves the problem around. It's not a goal that anything in the standard Python library cater to bad thread practice here (the bad thread practice being, again, starting a thread as a side effect of importing). -- http://mail.python.org/mailman/listinfo/python-list
