Robert Xiao added the comment:
I think this _is_ a bug. Most of the itertools are quite thrifty with memory,
and exceptions are explicitly documented.
The itertools generally only require memory proportional to the number of
iterations consumed. However, `itertools.product` requires an enormous up-front
memory cost even if only a few iterations are performed. There are good reasons
to extract only a few iterations from a `product` - for example, a search
procedure where only the first item in the product is infinite:
for year, day in product(count(2010), range(365)):
if rare_event(year, day):
break
Or, in an application I recently tried, to count occurrences of something up to
a limit:
prod = product(count(), range(n))
filtered = ifilter(pred, prod)
want = list(islice(filtered, 101))
if len(want) > 100:
print "Too many matches"
Having `product` greedily consume all of its input iterators is rather
antithetical to the notion of itertools being lazy generators, and it breaks
efficient composability with the other itertools.
The fix should be fairly straightforward: like tee, maintain lists of items
already generated as the input iterators are exhausted, and add a note that
"product"'s memory usage may slowly grow over time (as opposed to growing to
maximum size immediately).
----------
nosy: +nneonneo
_______________________________________
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue10109>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
