Addendum to my example:
If my get_tcp_header was made a class it would also be possible for it to
support the `__advance__` protocol:
```
class TCPHeaderIter(Iterator[TCPHeader]):
def __init__(self, stream: Iterator[Byte]):
self.stream = stream
def __next__(self) -> TCPHeader:
# similar to the body of the while loop
def __advance__(self, n):
for _ in range(n):
self.stream.advance(2)
total_length = ...
self.stream.advance(total_length - 4)
```
Now I don't have a use case for `__advance__` on a TCP header iterator but
one might
want to sample every N'th header.
On Wed, Oct 7, 2020 at 3:06 PM Caleb Donovick <donov...@cs.stanford.edu>
wrote:
> > For `__advance__` to be an official Python protocol, it would almost
> > certainly have to be of use for *general purpose iterators*, not just
> > specialised ones -- and probably not *hypothetical* iterators which may
> > not even exist. Do you have concrete examples of your skip list and tree
> > iterators that are in wide-spread use?
>
> "I am +0.3 on this as I don't personally have a need"
> " (I'm not sure why you would ever want to `__advance__` a tree iterator)"
>
> > What's your use case for advancing a count object, rather than just
> > creating a new one?
>
> Itertools.count was an example (hence the use of "e.g.") of an iterator
> which can be efficiently
> advanced without producing intermediate state. Clearly anyone can advance
> it manually.
> My point is that an iterator may have an efficient way to calculate its
> state some point in the future
> without needing to calculate the intermediate state. For example the
> fibonacci sequence has a closed
> form formula for the nth element and hence could be advanced efficiently.
>
> I realize my original examples were contrived, but I have a better one.
> Consider the task of collecting TCP headers from an iterator of bytes:
>
> ```
> def get_tcp_headers(stream: Iterator[Byte]):
> while stream:
> # Move to the total length field of the IP header
> stream.advance(2)
> # record the total length (two bytes)
> total_length = ...
> # skip the rest of IP header
> stream.advance(28)
> # record the TCP header
> header = ...
> yield header
> stream.advance(total_length - 32 - len(header))
> ```
>
> Maybe Kevin can tell us what motivated him to post this idea but I can see
> many places in parsing where you might want
> to skip arbitrary portions of a stream. The beauty of iterators is you
> don't need to be concerned with the underlying data
> structure. Ideally I shouldn't need to write two versions of some parse
> function one which operates on sequences and one
> that operates on iterables, just so I can efficiently `advance` the
> sequences.
>
> -- Caleb Donovick
>
> On Tue, Oct 6, 2020 at 6:16 PM Steven D'Aprano <st...@pearwood.info>
> wrote:
>
>> On Tue, Oct 06, 2020 at 02:27:54PM -0700, Caleb Donovick wrote:
>> > I am +0.3 on this as I don't personally have a need for this but do see
>> the
>> > utility.
>> >
>> > I can think of a number of examples where an `__advance__` would be
>> > preferable to any of the proposed solutions:
>> [...]
>>
>> For `__advance__` to be an official Python protocol, it would almost
>> certainly have to be of use for *general purpose iterators*, not just
>> specialised ones -- and probably not *hypothetical* iterators which may
>> not even exist. Do you have concrete examples of your skip list and tree
>> iterators that are in wide-spread use?
>>
>> Specialised iterators can create whatever extra APIs they want to
>> support, but the official iterator protocol intentionally has a very
>> basic API:
>>
>> - anything with an `__iter__` method which returns itself;
>> - and a `__next__` method that returns the next value, raising
>> StopIteration when exhausted.
>>
>> This is a bare minimum needed to make an iterator, and we like it that
>> way. For starters, it means that generators are iterators.
>>
>> If people want to supply objects that support the iterator protocol
>> but also offer a rich API including:
>>
>> - peek
>> - restart
>> - previous
>> - jump ahead (advance)
>>
>> all features that have been proposed, there is nothing stopping you from
>> adding those features to your iterator classes. But they all have
>> problems if considered to be necessary for *all* iterators.
>>
>> I would expect that, given a sufficiently compelling real-world
>> use-case, we would be prepared to add a jump ahead method to
>> list-iterators, as a specific feature of that iterator, not of all
>> iterators.
>>
>>
>> > A skip list which doesn't support O(1) random access but can advance
>> faster
>> > than naively calling next repeatedly
>> > A lazy infinite iterator which can efficiently calculate its state at
>> some
>> > future point (e.g. `itertools.count`)
>>
>> What's your use case for advancing a count object, rather than just
>> creating a new one?
>>
>> it = itertools.count() # start at 0
>> process(it) # process some values
>> it.advance(state) # jump forward
>> process(it) # process some more values
>>
>> as opposed to what is already possible:
>>
>> it = itertools.count()
>> process(it)
>> it = itertools.count(state)
>> process(it)
>>
>> Real-world use-cases for this feature are far more useful than contrived
>> and artifical use-cases unlikely to ever occur in real code.
>>
>>
>> > My ladder two examples demonstrate that this could have utility outside
>> of
>> > sequences but for iterators in general.
>>
>> I'm sorry, I don't know what your ladder two examples are. Did you post
>> them in another thread?
>>
>>
>> --
>> Steve
>> _______________________________________________
>> Python-ideas mailing list -- python-ideas@python.org
>> To unsubscribe send an email to python-ideas-le...@python.org
>> https://mail.python.org/mailman3/lists/python-ideas.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-ideas@python.org/message/I6PN73GZZ5K3L5Z4F6DQWOVHRZ2IMFMD/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
_______________________________________________
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at
https://mail.python.org/archives/list/python-ideas@python.org/message/ZMCIDZPAGZTIWRHSVUDTDWTLQQ7YXJVW/
Code of Conduct: http://python.org/psf/codeofconduct/
