New submission from Torsten Landschoff:
The behaviour of multiprocessing.Queue surprised me today in that Queue.get()
may raise an exception even if an item is immediately available. I tried to
flush entries without blocking by using the timeout=0 keyword argument:
$ /opt/python3/bin/python3
Python 3.4.0b1 (default:247f12fecf2b, Jan 6 2014, 14:50:23)
[GCC 4.6.3] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from multiprocessing import Queue
>>> q = Queue()
>>> q.put("hi")
>>> q.get(timeout=0)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/opt/python3/lib/python3.4/multiprocessing/queues.py", line 107, in get
raise Empty
queue.Empty
Actually even passing a small non-zero timeout will not give me my queue entry:
>>> q.get(timeout=1e-6)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/torsten/opensrc/cpython/Lib/multiprocessing/queues.py", line 107,
in get
raise Empty
queue.Empty
Expected behaviour for me would be to return the item that is in the queue. I
know that there is a kwarg *block* which gives me the desired behaviour:
>>> q.get(block=False)
'hi'
In my case the get call is embedded in my own module which does not currently
expose the block parameter. My local solution is of course to update the
wrapper:
if timeout == 0:
timeout = None
block = False
However I see a few smells here in the python standard library. First,
everything else seems to accept timeout=0 as nonblocking:
>>> import threading
>>> lock = threading.Lock()
>>> lock.acquire(timeout=0)
True
>>> from queue import Queue
>>> q = Queue()
>>> q.put("hi")
>>> q.get(timeout=0)
'hi'
Of special note is that queue.Queue behaves as I would have expected. IMHO it
should be consistent with multiprocessing.Queue.
Also note that queue.Queue.get() and queue.Queue.put() name their blocking flag
"block", while everybody else uses "blocking".
As a side note, I think the current approach is flawed in computing the
deadline. Basically it does the following:
deadline = time.time() + timeout
if not self._rlock.acquire(block, timeout):
raise Empty
timeout = deadline - time.time()
if timeout < 0 or not self._poll(timeout):
raise Empty
On my system, just taking the time twice and computing the delta takes 2
microseconds:
>>> import time
>>> t0 = time.time(); time.time() - t0
2.384185791015625e-06
Therefore calling Queue.get(block, timeout) with 0 < timeout < 2e-6 will never
return anything from the queue even though Queue.get(block=False) would do
that. This contradicts the idea that Queue.get(block=False) will return faster
than with block=True with any timeout > 0.
Apart from that, as Python does not currently support waiting on multiple
sources, we currently often check a queue with a small timeout concurrently
with doing other stuff. In case the system get really loaded, I would expect
this to cause problems because the updated timeout may fall below zero.
Suggested patch attached.
----------
components: Library (Lib)
files: queue_timeout_0.diff
keywords: patch
messages: 207443
nosy: torsten
priority: normal
severity: normal
status: open
title: multiprocessing.Queue.get() raises queue.Empty exception if even if an
item is available
type: behavior
versions: Python 2.7, Python 3.1, Python 3.2, Python 3.3, Python 3.4, Python 3.5
Added file: http://bugs.python.org/file33327/queue_timeout_0.diff
_______________________________________
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue20147>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
