[issue19993] Pool.imap doesn't work as advertised

[Josh Rosenberg]

Josh Rosenberg  added the comment:

Related: issue29842 "Make Executor.map work with infinite/large inputs 
correctly" for a similar problem in concurrent.futures (but worse, since it 
doesn't even allow you to begin consuming results until all inputs are 
dispatched).

A similar approach to my Executor.map patch could probably be used with 
imap/imap_unordered.

--
nosy: +josh.r

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19993] Pool.imap doesn't work as advertised

[Ned Deily]

Change by Ned Deily :


--
nosy: +davin, pitrou

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19993] Pool.imap doesn't work as advertised

[Alex Garel]

Alex Garel  added the comment:

Hello, I think this is a really important feature, it hits me hard those days. 

It would also solve https://bugs.python.org/issue19173 in a nice way.

--
nosy: +alex-garel

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19993] Pool.imap doesn't work as advertised

[Jurjen N.E. Bos]

New submission from Jurjen N.E. Bos:

The pool.imap and pool.imap_unordered functions are documented as a lazy 
version of Pool.map.
In fact, they aren't: they consume the iterator argument as a whole. This is 
almost certainly not what the user wants: it uses unnecessary memory and will 
be slower than expected if the output iterator isn't consumed in full. In fact, 
there isn't much use at all of imap over map at the moment.
I tried to fixed the code myself, but due to the two-level queueing of the 
input arguments this is not trivial.
Stackoverflow's Blckknght wrote a simplified solution that gives the idea how 
it should work.
Since that wasn't posted here, I thought it would be useful to put it here, 
even if only for documentation purposes.

--
components: Library (Lib)
files: mypool.py
messages: 206279
nosy: jneb
priority: normal
severity: normal
status: open
title: Pool.imap doesn't work as advertised
type: behavior
Added file: http://bugs.python.org/file33158/mypool.py

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19993
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19993] Pool.imap doesn't work as advertised

[Tim Peters]

Tim Peters added the comment:

Nice to see you, Jurjen!  Been a long time :-)

I'd like to see changes here too.  It's unclear what a lazy version  is 
intended to mean, exactly, but I agree the actual behavior is surprising, and 
that mpool.py is a lot less surprising in several ways.

I got bitten by this just last week, when running a parallelized search over a 
massive space _expected_ to succeed after exploring a tiny fraction of the 
search space.  Ran out of system resources because imap_unordered() tried to 
queue up countless millions of work descriptions.  I had hoped/expected that it 
would interleave generating and queue'ing a few inputs with retrieving 
outputs, much as mpool.py behaves.

In that case I switched to using apply_async() instead, interposing my own 
bounded queue (a collections.deque used only in the main program) to throttle 
the main program.  I'm still surprised it was necessary ;-)

--
nosy: +tim.peters

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19993
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19993] Pool.imap doesn't work as advertised

[Tim Peters]

Tim Peters added the comment:

Just for interest, I'll attach the worm-around I mentioned (imu.py).  At this 
level it's a very simple implementation, but now that I look at it, it's 
actually a lazy implementation of imap() (or of an unimaginative ;-) 
imap_unordered()).

--
Added file: http://bugs.python.org/file33166/imu.py

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19993
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com