[issue12897] Support for iterators in multiprocessing map
Richard Oudkerk shibt...@gmail.com added the comment: Unless you have a reason why imap() does not solve the problem I will eventually close the issue as rejected. -- resolution: - rejected stage: - committed/rejected status: open - pending ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12897 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12897] Support for iterators in multiprocessing map
andrew cooke and...@acooke.org added the comment: hi - i'm the original author (may be using a different account). as far as i remember, i raised this because it seemed relevant given the link i gave. if you've looked at the issue and think your approach would work, or that this should be closed, or whatever, that's fine by me. i'm not going to check myself - i can't remember anything about this now (nearly a year later) and it's not my place to worry about your code (no offence - just trying to clarify that i have no skin in this game). -- status: pending - open ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12897 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12897] Support for iterators in multiprocessing map
Richard Oudkerk shibt...@gmail.com added the comment: I'll close then. -- status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12897 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12897] Support for iterators in multiprocessing map
Richard Oudkerk shibt...@gmail.com added the comment: If you want lazy operation then you should use imap(f, it[, chunksize]) rather than using map_async(f, it). This will return an iterator rather than a list. Also, the iterator's next() method has a timeout argument. (chunksize is the number of items which get sent to a worker in each batch, with 1 being the default.) -- nosy: +sbt ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12897 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12897] Support for iterators in multiprocessing map
New submission from andrew cooke and...@acooke.org: http://stackoverflow.com/questions/7306522/combining-itertools-and-multiprocessing/7307078 suggests (and the idea itself seems reasonable) that it would sometimes be useful for multiprocessing to operate correctly (ie lazily) with lazy input (iterables). for example, if the input is large, or perhaps generated by some other process on demand. obviously this complicates matters, given the asynchronous nature of a worker pool, and would mean re-allocating the results list as required. but in principle i suspect it would be possible and might be a useful extension. -- components: Library (Lib) messages: 143511 nosy: acooke priority: normal severity: normal status: open title: Support for iterators in multiprocessing map type: feature request versions: Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12897 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12897] Support for iterators in multiprocessing map
Antoine Pitrou pit...@free.fr added the comment: Since it's a feature request, I would suggest to look whether it can apply to concurrent.futures instead. -- nosy: +pitrou versions: +Python 3.3 -Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12897 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
