Thanks for the patch. I think I understand most of it now. Have you written and tested the Tornado conversion yet? What does it look like?
Now let's go back to the Deferred converter you showed. The asyncio converter would be very similar, just calling f.set_exception() instead of f.set_exc_info(). So support I have an asyncio coroutine and I yield a Deferred. I'm probably not calling Deferred() myself -- I'm calling some Twisted thing that happens to return a Deferred (which means that the result isn't ready yet -- but at some point it will be ready and then the deferred will fire). So I yield this Deferred, and the asyncio event loop continues. But there seems a piece missing here: we need the Twisted event loop to run in order for the Deferred to make progress. (Presumably the Deferred waits for I/O completion on some socket -- and that socket is registered with the Twisted event loop, not with the Tulip event loop.) I assume there's a way this is done in Tornado and an analogous way should work for Tulip too. But what is it? Is this part in your Tornado patch too? On Sun, Jan 25, 2015 at 5:19 PM, Ben Darnell <b...@bendarnell.com> wrote: > On Sun, Jan 25, 2015 at 6:11 PM, Guido van Rossum <gu...@python.org> > wrote: > >> On Sun, Jan 25, 2015 at 2:52 PM, Ben Darnell <b...@bendarnell.com> wrote: >> >>> On Sun, Jan 25, 2015 at 5:31 PM, Guido van Rossum <gu...@python.org> >>> wrote: >>> >>>> I'm probably a bit dense today, but without browsing the code I'm not >>>> entirely sure how your thing works. >>>> >>> >>> Sorry, the implementation in Tornado is in this series of commits: >>> https://github.com/tornadoweb/tornado/compare/05c3073ce363...841b2d4de32b >>> >> >> OK, not light Sunday afternoon reading. :-) >> > > It's not as bad as it looks; there's some refactoring and > backwards-compatibililty weirdness in there too. The asyncio patch will be > easier to follow. > > >> >> >>> What type of thing is the single-dispatch dispatching on? >>>> >>> >>> It dispaches on the type of the thing that was yielded. So to handle >>> Deferred, I have this: >>> >>> + @gen.convert_yielded.register(Deferred) >>> + def _(d): >>> + f = Future() >>> + def errback(failure): >>> + try: >>> + failure.raiseException() >>> + # Should never happen, but just in case >>> + raise Exception("errback called without error") >>> + except: >>> + f.set_exc_info(sys.exc_info()) >>> + d.addCallbacks(f.set_result, errback) >>> + return f >>> >> >> OK, I think I get this part. I'm so glad I invested a week of my summer >> vacation two years ago in understanding Deferred >> <https://groups.google.com/d/msg/python-tulip/ut4vTG-08k8/PWZzUXX9HYIJ>. >> :-) >> >> >>> I then call tornado.gen.convert_yielded() on the result of g.send() (or >>> g.throw()) before doing anything else with it. >>> >>> >>>> How would the asyncio and Tornado versions cooperate? >>>> >>> >>> They don't have to cooperate; they can be completely unaware of each >>> other. I was able to implement this for Tornado without asyncio or Twisted >>> having anything comparable. >>> >> >> Ah, so we each just register things that handle the other's kind of >> Future by converting it to one of our own kind, and then the framework of >> the caller (the generator that calls yield) gets to keep control. Right? >> I'm a little worried (again not yet having fully internalized the whole >> process) that there might be some cases where frameworks keep passing the >> ball without making progress, but I can also easily imagine that that's not >> how it works. >> > > Right. The coroutine runner (tornado.gen.coroutine, asyncio.Task, > twisted.internet.defer.inlineCallbacks) is a constant; a Tornado coroutine > doesn't magically become an asyncio.Task. It's just the Futures that get > wrapped. There's no risk of getting stuck in a loop - convert_yielded() is > only called once and if it doesn't return a value of the right Future type > then it's an error. > > >> >> >>> >>> >>>> >>>> Perhaps you can submit a patch for asyncio to clarify the proposal? >>>> >>> >>> Sure. Is the tulip project on code.google.com still the place for that >>> or is new development happening in the cpython repo? >>> >> >> It doesn't really matter -- we merge back and forth these days. You may >> find the asyncio repo easier to deal with. >> > > Indeed. Here's a patch: https://codereview.appspot.com/193540043/ > > There are two subtleties here: > * asyncio requires the use of 'yield from' everywhere, but third-party > future-like objects may not be iterable, so they must be allowed with > 'yield'. I disabled a check for Future._blocking when convert_yielded > returned anything but the original object > * asyncio.Future requires an event loop to be set for the current thread > at construction time. This required me to call asyncio.set_event_loop in > the test for this feature when most comparable tests in test_tasks.py do > not have this. > > -Ben > > >> >>> -Ben >>> >>> >>>> >>>> On Sun, Jan 25, 2015 at 9:42 AM, Ben Darnell <b...@bendarnell.com> >>>> wrote: >>>> >>>>> In Tornado 4.1 [1], I've added a functools.singledispatch-based >>>>> registry for objects that can be yielded in coroutines. This allows for >>>>> Future-like objects from different frameworks to be mixed seamlessly in >>>>> the >>>>> same coroutine (We currently have support for tornado.concurrent.Future, >>>>> concurrent.futures.Future, asyncio.Future[2], and >>>>> twisted.internet.defer.Deferred) >>>>> >>>>> I'd like to suggest that asyncio add something similar to facilitate >>>>> the use of libraries from different frameworks in the same coroutine, not >>>>> just sharing the same event loop. It would be a pretty simple change: just >>>>> call a function decorated with @singledispatch (whose default >>>>> implementation is the identity function) before handling the result in >>>>> Task._step. >>>>> >>>>> I've included a simple example of a hybrid tornado/asyncio coroutine >>>>> below. >>>>> >>>>> -Ben >>>>> >>>>> [1] Currently in beta, installable with `pip install >>>>> https://github.com/tornadoweb/tornado/archive/v4.1.0b1.zip` >>>>> <https://github.com/tornadoweb/tornado/archive/v4.1.0b1.zip> >>>>> [2] The future objects are all similar enough that they don't really >>>>> need this level of abstraction, but they serve as a proof of concept. >>>>> >>>>> # requirements: >>>>> # python3.4 (for 3.3, add asyncio and singledispatch) >>>>> # aiohttp >>>>> # https://github.com/tornadoweb/tornado/archive/v4.1.0b1.zip >>>>> import aiohttp >>>>> import asyncio >>>>> import tornado.gen >>>>> import tornado.httpclient >>>>> import tornado.ioloop >>>>> import tornado.platform.asyncio >>>>> >>>>> @tornado.gen.coroutine >>>>> def main(): >>>>> t_client = tornado.httpclient.AsyncHTTPClient() >>>>> t_response = yield t_client.fetch('http://www.google.com') >>>>> print('tornado: read %d bytes with status code %d' % >>>>> (len(t_response.body), t_response.code)) >>>>> >>>>> a_response = yield from aiohttp.request('GET', ' >>>>> http://www.google.com') >>>>> a_body = yield from a_response.read() >>>>> print('aiohttp: read %d bytes with status code %d' % >>>>> (len(a_body), a_response.status)) >>>>> >>>>> # Alternate python2-compatible syntax >>>>> a_response2 = yield asyncio.async(aiohttp.request('GET', ' >>>>> http://www.google.com')) >>>>> a_body2 = yield asyncio.async(a_response2.read()) >>>>> print('aiohttp2: read %d bytes with status code %d' % >>>>> (len(a_body2), a_response2.status)) >>>>> >>>>> if __name__ == '__main__': >>>>> >>>>> tornado.ioloop.IOLoop.configure(tornado.platform.asyncio.AsyncIOMainLoop) >>>>> tornado.ioloop.IOLoop.current().run_sync(main) >>>>> >>>>> >>>> >>>> >>>> -- >>>> --Guido van Rossum (python.org/~guido) >>>> >>> >>> >> >> >> -- >> --Guido van Rossum (python.org/~guido) >> > > -- --Guido van Rossum (python.org/~guido)
