[issue4999] multiprocessing.Queue does not order objects

Change by Irit Katriel : -- resolution: -> out of date stage: -> resolved status: pending -> closed ___ Python tracker ___ ___

[issue4999] multiprocessing.Queue does not order objects

Irit Katriel added the comment: I see the documentation mentions it now (in the second gray box here: https://docs.python.org/3.8/library/multiprocessing.html#pipes-and-queues ) and it also suggests to use a shared queue if this is an issue. Any objections to closing this issue?

[issue4999] multiprocessing.Queue does not order objects

Changes by Jean-Paul Calderone jean-p...@hybridcluster.com: -- nosy: -exarkun ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4999 ___ ___

[issue4999] multiprocessing.Queue does not order objects

Changes by Stefan Krah stefan-use...@bytereef.org: -- nosy: -skrah ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4999 ___ ___ Python-bugs-list

[issue4999] multiprocessing.Queue does not order objects

Changes by Ned Deily n...@acm.org: -- nosy: +sbt ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4999 ___ ___ Python-bugs-list mailing list

[issue4999] multiprocessing.Queue does not order objects

Ask Solem a...@opera.com added the comment: Updated doc patch -- nosy: +asksol Added file: http://bugs.python.org/file19350/issue-4999.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4999

[issue4999] multiprocessing.Queue does not order objects

Georg Brandl ge...@python.org added the comment: Are you sure this needs to be a warning? We try to use them very sparingly. (Also note the 3-space indent for reStructuredText markup.) -- nosy: +georg.brandl ___ Python tracker

[issue4999] multiprocessing.Queue does not order objects

Stefan Praszalowicz deubeul...@gmail.com added the comment: I just got surprised by this, and I agree that updating the doc would be nice, because as of now, it states quite explicitly that the Queue and JoinableQueue types are multi-producer, multi-consumer FIFO queues. -- nosy:

[issue4999] multiprocessing.Queue does not order objects

Stefan Krah stefan-use...@bytereef.org added the comment: I think it would be nice to update the documentation if this isn't resolved yet. The patch adds a warning that FIFO behavior is not guaranteed. -- keywords: +patch nosy: +skrah Added file:

[issue4999] multiprocessing.Queue does not order objects

Changes by Gabriel Genellina gagsl-...@yahoo.com.ar: -- nosy: +gagenellina ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4999 ___ ___

[issue4999] multiprocessing.Queue does not order objects

Changes by Jesse Noller jnol...@gmail.com: -- priority: - normal ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4999 ___ ___ Python-bugs-list

[issue4999] multiprocessing.Queue does not order objects

Antoine Pitrou pit...@free.fr added the comment: Rather than providing strict ordering of Queue input/output, perhaps a separate or derived class (PriorityQueue?) should be provided just for that matter (heapq-based?). -- nosy: +pitrou ___ Python

[issue4999] multiprocessing.Queue does not order objects

Raymond Hettinger rhettin...@users.sourceforge.net added the comment: The Queue module design should be followed as closely as possible. It provides FIFO order as the default, with options for LIFO and priority order. -- nosy: +rhettinger ___ Python

[issue4999] multiprocessing.Queue does not order objects

Jesse Noller jnol...@gmail.com added the comment: I agree, at very least it should provide the same default behavior that Queue does, however, it is not obvious to me how to resolve this in mp.Queue ___ Python tracker rep...@bugs.python.org

[issue4999] multiprocessing.Queue does not order objects

Antoine Pitrou pit...@free.fr added the comment: That's why I was suggesting to leave Queue as it is and provide a separate PriorityQueue. With a PriorityQueue the user is free to put (timestamp, object) tuples in it if he wants a strict chronologic ordering. He can also use sequence numbers

[issue4999] multiprocessing.Queue does not order objects

Jean-Paul Calderone exar...@divmod.com added the comment: Jesse, can you explain the cause of the bug? Maybe that will inspire someone to come up with an idea for a fix. -- nosy: +exarkun ___ Python tracker rep...@bugs.python.org

[issue4999] multiprocessing.Queue does not order objects

Jesse Noller jnol...@gmail.com added the comment: As a note to myself: adding a simple print to the multiprocessing.Queue put method, I can see the calls occurring from the children in order, for example: obj: 0.025193 [proc1] Got lock obj: 0.227725 [proc1] Released lock obj: 0.228401

[issue4999] multiprocessing.Queue does not order objects

Jesse Noller jnol...@gmail.com added the comment: On a put to the queue, the object is appended onto a deque (self._buffer) - this buffer is managed by a thread (_start_thread) which is handed the _feed method as the target. I suspect the bug is actually in the _feed method.

[issue4999] multiprocessing.Queue does not order objects

New submission from Frédéric Sagnes speedup+pyt...@gmail.com: Objects contained in a multiprocessing.Queue object are not comming out of the queue in the same order as they went in. For instance, if I put in object1, object2 and object3 in this very time sequence from multiple processes, they

[issue4999] multiprocessing.Queue does not order objects

Changes by Jesse Noller jnol...@gmail.com: -- assignee: - jnoller nosy: +jnoller ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4999 ___ ___