Steven Bethard wrote: > def filterdups(iterable): > seen = set() > for item in iterable: > if item not in seen: > seen.add(item) > yield item > > Adding this to, say, itertools would cover all my use cases. And as > long as you don't have too many duplicates, filterdups as above should > keep memory consumption down better.
Thinking about this further - memory usage would be almost identical. By the time you completed the iterable, you would have built up exactly the same set internally - although probably not as memory efficient since it would be being built piecemeal. OTOH, an ordered set has a bit of extra memory for maintaining the order, so it's going to be pretty close. The only thing this gains you (and it's significant) is the ability to work on any iterable lazily. Tim Delaney _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
