[issue17025] reduce multiprocessing.Queue contention

Charles-François Natali added the comment: Committed, thanks! -- resolution: -> fixed stage: commit review -> committed/rejected status: open -> closed ___ Python tracker ___ __

[issue17025] reduce multiprocessing.Queue contention

Roundup Robot added the comment: New changeset 5022ee7e13a2 by Charles-François Natali in branch 'default': Issue #17025: multiprocessing: Reduce Queue and SimpleQueue contention. http://hg.python.org/cpython/rev/5022ee7e13a2 -- ___ Python tracker

[issue17025] reduce multiprocessing.Queue contention

Roundup Robot added the comment: New changeset bedb4cbdd311 by Charles-François Natali in branch 'default': Issue #17025: Add dumps() and loads() to ForkingPickler. http://hg.python.org/cpython/rev/bedb4cbdd311 -- nosy: +python-dev ___ Python tracker

[issue17025] reduce multiprocessing.Queue contention

Richard Oudkerk added the comment: On 24/03/2013 12:16pm, Charles-François Natali wrote: > The object is overwritten by the pickled data, so it's not necessary > anymore, no? Yes, you are right. -- ___ Python tracker

[issue17025] reduce multiprocessing.Queue contention

Charles-François Natali added the comment: > The old code deleted the obj in the feeder thread as soon as it was sent at > lines 247 and 253 -- see Issue #16284. I think that should be retained. The object is overwritten by the pickled data, so it's not necessary anymore, no? --

[issue17025] reduce multiprocessing.Queue contention

Richard Oudkerk added the comment: The old code deleted the obj in the feeder thread as soon as it was sent at lines 247 and 253 -- see Issue #16284. I think that should be retained. Apart from that LGTM. -- ___ Python tracker

[issue17025] reduce multiprocessing.Queue contention

Charles-François Natali added the comment: I'm splitting the patches: - one which adds loads and dumps to ForkingPicler - the contention reduction patch I'd like to commit them soon. -- Added file: http://bugs.python.org/file29559/queues_contention.diff Added file: http://bugs.python.or

[issue17025] reduce multiprocessing.Queue contention

Charles-François Natali added the comment: > By the way, I forgot to mention it previously, but > multiprocessing.connection uses a custom pickler (ForkingPickler). Thanks, I didn't know. Here's a patch using ForkingPickler. I did a bit of refactoring to move the pickling code from connection.p

[issue17025] reduce multiprocessing.Queue contention

Antoine Pitrou added the comment: By the way, I forgot to mention it previously, but multiprocessing.connection uses a custom pickler (ForkingPickler). By replacing it with plain pickle.dumps() calls, you may produce regressions since some types won't be sendable anymore (although the test sui

[issue17025] reduce multiprocessing.Queue contention

Changes by Charles-François Natali : -- stage: -> commit review versions: +Python 3.4 Added file: http://bugs.python.org/file29528/queues_contention-1.diff ___ Python tracker __

[issue17025] reduce multiprocessing.Queue contention

Charles-François Natali added the comment: > No. I only looked at the diff and assumed both changes were for Queue. OK, great. > Since you marked issue 10886 as superceded, do you intend to do the > pickling in put()? Actually no, I'll reopen it. I find the performance hit important, though.

[issue17025] reduce multiprocessing.Queue contention

Richard Oudkerk added the comment: On 04/03/2013 8:01pm, Charles-François Natali wrote: >> It looks like queues_contention.diff has the line >> >> obj = pickle.dumps(obj) >> >> in both _feed() and put(). Might that be why the third set of benchmarks >> was slower than the second? > > _feed(

[issue17025] reduce multiprocessing.Queue contention

Charles-François Natali added the comment: > It looks like queues_contention.diff has the line > > obj = pickle.dumps(obj) > > in both _feed() and put(). Might that be why the third set of benchmarks > was slower than the second? _feed() is a Queue method, put() its SimpleQueue() counterpar

[issue17025] reduce multiprocessing.Queue contention

Richard Oudkerk added the comment: It looks like queues_contention.diff has the line obj = pickle.dumps(obj) in both _feed() and put(). Might that be why the third set of benchmarks was slower than the second? -- ___ Python tracker

[issue17025] reduce multiprocessing.Queue contention

Antoine Pitrou added the comment: IMHO the simple version is good enough. -- ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue17025] reduce multiprocessing.Queue contention

Charles-François Natali added the comment: So, what do you think? Is the simple version offloading the serialization to queue enough, or should we go for a full-blown atomic Connection/Pipe/etc? I find the performance gain quite appreciable (basically queue didn't scale at all, now it scales wit

[issue17025] reduce multiprocessing.Queue contention

Charles-François Natali added the comment: > For the record, I tried the Connection approach and here is what I ended up > with. I don't really like the API. Having to pass an external lock is IMO a bad idea, it should be a private instance field. Also, for consistency we'd probably need send_b

[issue17025] reduce multiprocessing.Queue contention

Antoine Pitrou added the comment: For the record, I tried the Connection approach and here is what I ended up with. -- Added file: http://bugs.python.org/file28873/locked_send_recv.patch ___ Python tracker ___

[issue17025] reduce multiprocessing.Queue contention

Charles-François Natali added the comment: > I would like to suggest again my idea of doing it in Connection instead, > with new methods (e.g. locked_send and locked_recv). Especially given > it can be useful in user code to have a thread-safe Connection (I'm in > this situation currently). I in

[issue17025] reduce multiprocessing.Queue contention

Antoine Pitrou added the comment: > Here's an implementation of the idea posted on python-ideas > (http://mail.python.org/pipermail/python-ideas/2013-January/018846.html). > > The principle is really simple, we just serialize/unserialize the > objects before/after holding the locks. This leads t

[issue17025] reduce multiprocessing.Queue contention

Changes by Charles-François Natali : Added file: http://bugs.python.org/file28813/multi_queue.py ___ Python tracker ___ ___ Python-bugs-list m

[issue17025] reduce multiprocessing.Queue contention

New submission from Charles-François Natali: Here's an implementation of the idea posted on python-ideas (http://mail.python.org/pipermail/python-ideas/2013-January/018846.html). The principle is really simple, we just serialize/unserialize the objects before/after holding the locks. This lead