On Tue, Mar 3, 2015 at 2:06 PM, Luciano Ramalho <luci...@ramalho.org> wrote:
> On Tue, Mar 3, 2015 at 1:49 AM, Guido van Rossum <gu...@python.org> wrote:
> > It's exceedingly subtle -- that's why the docstring contains an example
> of
> > how to use it.
> >
> > Note the final two lines:
> >
> > for _ in range(len(todo)):
> > yield _wait_for_one()
> >
> > This is yield, not yield-from. as_completed() is not a coroutine -- it
> is an
> > iterator and the caller must loop over it. Each time the caller gets one
> > thing from the loop, what is that thing? It's _wait_for_one() -- i.e.
> it's a
> > coroutine! Then the caller has to use "yield from" on that coroutine,
> which
> > will then wait until a future emerges from the queue, and then
> > _wait_for_one() either returns that future's result or raises its
> exception.
> > This is also where timeouts are processed (they result in a dummy None
> > emerging from the queue).
>
> Thank you for this, Guido! I can now appreciate the subtle wording of
> "Return an iterator whose values, when waited for, are Future
> instances."
>
> Is there a name for these _wait_for_one generators that yield a
> Future? Giving those objects a good name could make it easier to
> explain.
>
Well, as_completed is a generator; generators yield a stream of results,
and in this case each result is itself a coroutine (to be precise, a
coroutine object, not a coroutine function -- _wait_for_one is the latter,
_wait_for_one() is the former).
> As it stands, the note is misleading: "The futures f are not
> necessarily members of fs." The f you get from as_completed is not a
> Future, but that thing-you-yield-from-to-get-a-Future. Without giving
> that thing a name, the note could say "The future yielded from each f
> is not necessarily a member of fs."
>
Yeah, this is a general concept, "either a coroutine or a future" and it
doesn't really have a name. It's also known informally as "the kind of
thing you pass to yield from", which isn't so helpful either. :-)
> > Hope this helps. The input futures never appear in the output, so the
> > warning in the docstring is an understatement (but don't remove it -- a
> > future implementation might use an optimized path if some of the futures
> are
> > *already* done.
>
> Do you mean it's possible some day as_completed() may return either
> things-you-yield-from-to-get-a-Future or actual Futures, i.e.
> instances of asyncio.Future?
>
Yes, that's totally possible. And those Futures could either be amongst the
input Futures or new Futures.
A trivial optimization which would return some Futures amongst the input
would be to add something like this to the top:
readies = {f for f in todo if f.done()}
if readies:
todo -= readies
for f in readies: yield f
I think at some point I had that, but I think the use case is marginal (you
almost never call this when there's much of a chance that any of the
futures are already done), and I wasn't sure about timeouts and other
guarantees.
Also, since the input can also be coroutines-or-Futures, this loop might
still yield some Futures that aren't in the input (because they were made
up by the async() call: todo = {async(f, loop=loop) for f in set(fs)}. Also
note that fs may be an iterator -- it's iterated over exactly once (by the
set(fs) call here).
--
--Guido van Rossum (python.org/~guido)
