[issue4999] multiprocessing.Queue does not order objects

[Irit Katriel]

Change by Irit Katriel :


--
resolution:  -> out of date
stage:  -> resolved
status: pending -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4999] multiprocessing.Queue does not order objects

[Irit Katriel]

Irit Katriel  added the comment:

I see the documentation mentions it now (in the second gray box here:
https://docs.python.org/3.8/library/multiprocessing.html#pipes-and-queues )

and it also suggests to use a shared queue if this is an issue. 


Any objections to closing this issue?

--
nosy: +iritkatriel
status: open -> pending

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4999] multiprocessing.Queue does not order objects

[Jean-Paul Calderone]

Changes by Jean-Paul Calderone jean-p...@hybridcluster.com:


--
nosy:  -exarkun

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4999
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4999] multiprocessing.Queue does not order objects

[Stefan Krah]

Changes by Stefan Krah stefan-use...@bytereef.org:


--
nosy:  -skrah

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4999
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4999] multiprocessing.Queue does not order objects

[Ned Deily]

Changes by Ned Deily n...@acm.org:


--
nosy: +sbt

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4999
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4999] multiprocessing.Queue does not order objects

[Ask Solem]

Ask Solem a...@opera.com added the comment:

Updated doc patch

--
nosy: +asksol
Added file: http://bugs.python.org/file19350/issue-4999.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4999
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4999] multiprocessing.Queue does not order objects

[Georg Brandl]

Georg Brandl ge...@python.org added the comment:

Are you sure this needs to be a warning?  We try to use them very sparingly.

(Also note the 3-space indent for reStructuredText markup.)

--
nosy: +georg.brandl

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4999
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4999] multiprocessing.Queue does not order objects

[Stefan Praszalowicz]

Stefan Praszalowicz deubeul...@gmail.com added the comment:

I just got surprised by this, and I agree that updating the doc would be nice, 
because as of now, it states quite explicitly that the Queue and JoinableQueue 
types are multi-producer, multi-consumer FIFO queues.

--
nosy: +Stefan.P
type:  - behavior

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4999
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4999] multiprocessing.Queue does not order objects

[Stefan Krah]

Stefan Krah stefan-use...@bytereef.org added the comment:

I think it would be nice to update the documentation if this isn't
resolved yet. The patch adds a warning that FIFO behavior is not
guaranteed.

--
keywords: +patch
nosy: +skrah
Added file: http://bugs.python.org/file16276/warn_fifo.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4999
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4999] multiprocessing.Queue does not order objects

[Gabriel Genellina]

Changes by Gabriel Genellina gagsl-...@yahoo.com.ar:


--
nosy: +gagenellina

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4999
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4999] multiprocessing.Queue does not order objects

[Jesse Noller]

Changes by Jesse Noller jnol...@gmail.com:


--
priority:  - normal

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4999
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4999] multiprocessing.Queue does not order objects

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

Rather than providing strict ordering of Queue input/output, perhaps a
separate or derived class (PriorityQueue?) should be provided just for
that matter (heapq-based?).

--
nosy: +pitrou

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4999
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4999] multiprocessing.Queue does not order objects

[Raymond Hettinger]

Raymond Hettinger rhettin...@users.sourceforge.net added the comment:

The Queue module design should be followed as closely as possible.  It
provides FIFO order as the default, with options for LIFO and priority
order.

--
nosy: +rhettinger

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4999
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4999] multiprocessing.Queue does not order objects

[Jesse Noller]

Jesse Noller jnol...@gmail.com added the comment:

I agree, at very least it should provide the same default behavior that 
Queue does, however, it is not obvious to me how to resolve this in 
mp.Queue

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4999
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4999] multiprocessing.Queue does not order objects

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

That's why I was suggesting to leave Queue as it is and provide a
separate PriorityQueue. With a PriorityQueue the user is free to put
(timestamp, object) tuples in it if he wants a strict chronologic
ordering. He can also use sequence numbers instead of timestamps, etc.

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4999
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4999] multiprocessing.Queue does not order objects

[Jean-Paul Calderone]

Jean-Paul Calderone exar...@divmod.com added the comment:

Jesse, can you explain the cause of the bug?  Maybe that will inspire
someone to come up with an idea for a fix.

--
nosy: +exarkun

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4999
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4999] multiprocessing.Queue does not order objects

[Jesse Noller]

Jesse Noller jnol...@gmail.com added the comment:

As a note to myself: adding a simple print to the multiprocessing.Queue 
put method, I can see the calls occurring from the children in order, 
for example:

obj:  0.025193 [proc1] Got lock
obj:  0.227725 [proc1] Released lock
obj:  0.228401 [proc2] Got lock
obj:  0.430501 [proc2] Released lock
obj:  0.431082 [proc0] Got lock
obj:  0.633048 [proc0] Released lock
obj:  0.633723 [proc4] Got lock
obj:  0.835692 [proc4] Released lock
obj:  0.836262 [proc3] Got lock
obj:  1.038197 [proc3] Released lock
obj:  1.038753 [proc1] Got lock
obj:  1.239288 [proc1] Released lock
obj:  1.240079 [proc2] Got lock
obj:  1.440773 [proc2] Released lock
obj:  1.441577 [proc0] Got lock
obj:  1.642142 [proc0] Released lock
obj:  1.642959 [proc4] Got lock
obj:  1.843331 [proc4] Released lock
obj:  1.844178 [proc3] Got lock
obj:  2.044549 [proc3] Released lock

Also attaching the process-based script

Added file: http://bugs.python.org/file12836/nick.py

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4999
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4999] multiprocessing.Queue does not order objects

[Jesse Noller]

Jesse Noller jnol...@gmail.com added the comment:

On a put to the queue, the object is appended onto a deque (self._buffer) 
- this buffer is managed by a thread (_start_thread) which is handed the 
_feed method as the target. I suspect the bug is actually in the _feed 
method.

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4999
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4999] multiprocessing.Queue does not order objects

[Frédéric Sagnes]

New submission from Frédéric Sagnes speedup+pyt...@gmail.com:

Objects contained in a multiprocessing.Queue object are not comming out
of the queue in the same order as they went in. For instance, if I put
in object1, object2 and object3 in this very time sequence from multiple
processes, they can end up comming out of the queue as object2, object1
then object3 instead of the original order.

When using the threading module instead of multiprocessing everything is
fine.

The provided test script adds strings to the queue with timestamps.
These messages are not ordered by timestamp when they are printed. This
is an output of the test script with format [...@time] message:

[2...@00406] Got lock
[2...@02424] Released lock
[2...@02426] Got lock
[2...@04439] Released lock
[...]
[2...@16459] Released lock
[2...@16461] Got lock
[2...@18464] Got lock
[2...@18462] Released lock
[2...@20466] Released lock

Using print to print the message immediatly prints the messages in the
right order.

See this mailing-list thread for details:
http://groups.google.com/group/comp.lang.python/browse_thread/thread/11a5c4ce4ff4382d/033dcd3607eacbf9

--
components: Extension Modules
files: testthreads.py
messages: 80162
nosy: ndfred
severity: normal
status: open
title: multiprocessing.Queue does not order objects
versions: Python 2.6
Added file: http://bugs.python.org/file12793/testthreads.py

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4999
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4999] multiprocessing.Queue does not order objects

[Jesse Noller]

Changes by Jesse Noller jnol...@gmail.com:


--
assignee:  - jnoller
nosy: +jnoller

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4999
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com