Hi Nick,
On 05/05/17 08:29, Nick Coghlan wrote:
And then given the proposed str.splitgroups() on the one hand, and the
existing memoryview.cast() on the other, offering
itertools.itergroups() as a corresponding building block specifically
for working with streams of regular data would make sense to me -
that's a standard approach in time-division multiplexing protocols,
and it also shows up in areas like digital audio processing as well
(where you're often doing things like shuffling incoming data chunks
into FFT buffers)
It looks to me like your "itertools.itergroups()" is similar to
more_itertools.chunked() - with at least one obvious change, see below(*).
If anyone wants to persue this (or any itertools) enhancement, then
please be aware of the following thread (and in particular the message
being linked to - and the bug and discussion that it is replying to):
https://mail.python.org/pipermail/python-dev/2012-July/120885.html
I have been told off for bringing this up already, but I do it again in
direct response to your suggestion because it seems there is a bar to
getting something included in itertools and something like "chunked()"
has already failed to make it. The thing to do is probably to talk
directly to Raymond to see if there's an acceptable solution first
before too much work is put into something that may be rejected as being
too high level.
It may be that a C version of "more_itertools" for things which people
would find a speedup useful might be a solution (where the
more_itertools package defers to those built-ins if they exist on the
version of Python its executing on, otherwise uses its existing
implementation as a fallback). I am not suggesting implementing the
_whole_ of more_itertools in C - it's quite large now.
(*) I had implemented itertools.chunked in C before (also for audio
processing, as it happens) and one thing that I didn't like is the way
strings get unpacked:
>>> tuple(more_itertools.chunked("foo bar baz", 2))
(['f', 'o'], ['o', ' '], ['b', 'a'], ['r', ' '], ['b', 'a'], ['z'])
If the chunked/itergroups method checked for the presence of a
__chunks__ or similar dunder method in the source sequence which returns
an iterator, then the string class could efficiently yield substrings
rather than individual characters which then had to be wrapped in a list
or tuple (which I think is what you wanted itergroups() to do):
>>> tuple(itertools.chunked("foo bar baz", 2))
('fo', 'o ', 'ba', 'r ', 'ba', 'z')
Similarly, for objects which _represent_ a lot of data but do not
actually hold those data literally (for example, range objects or even
memoryviews), the returned chunks can also be representations of the
data (subranges or subviews) and not the actual rendered data. For
example, the existing:
>>> range(10)
range(0, 10)
>>> tuple(more_itertools.chunked(range(10), 3))
([0, 1, 2], [3, 4, 5], [6, 7, 8], [9])
becomes:
>>> tuple(more_itertools.chunked(range(10), 3))
(range(0, 3), range(3, 6), range(6, 9), range(9, 10))
Obviously, with those short strings and ranges one could argue that
there's no point, but the principle of doing it this way scales better
than the version that collects all of the data in lists - for things
like chunks of some sort of "view" object, you would still only have the
actual data stored once in the original object.
I suppose that one thing to consider is what happens when an iterator is
passed to the chunked() function. An iterator could have a __chunks__
method which returned chunks of the source sequence from the existing
point in the iteration, however the difference between such an iterator
and one that _doesn't_ have a __chunks__ method is that in the second
case the iterator would be consumed by the fall-back code which just
does what more_itertools.chunked() does now, but in the first it would not.
Perhaps there is a precedent for that particular edge case with
iterators in a different context.
Hope that helps,
E.
_______________________________________________
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
