[issue28639] inspect.isawaitable(native_coro) returns int
New submission from Justin Mayfield: Patch available on my github fork.. https://github.com/mayfield/cpython/commit/8d50cc61f42fa280d380e9c10c420c949a619bef -- components: asyncio messages: 280280 nosy: Justin Mayfield, gvanrossum, yselivanov priority: normal severity: normal status: open title: inspect.isawaitable(native_coro) returns int type: behavior versions: Python 3.5, Python 3.6, Python 3.7 ___ Python tracker <http://bugs.python.org/issue28639> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24837] await process.wait() does not work with a new_event_loop
Justin Mayfield added the comment: I agree with Guido. I spent a couple hours trying to debug some of my own code that turned out to be this issue. My use case is single threaded too. Perhaps I'm daft but I don't understand why the child watcher is part of the event loop policy. At first glance it appears that it would be better suited being attached to any event loop in the main thread and then this wouldn't be an issue. I wonder if this design is intended to support launching subprocesses from non-main thread event loops, which seems inherently dangerous and potentially misguided to me. ------ nosy: +Justin Mayfield ___ Python tracker <http://bugs.python.org/issue24837> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError
Justin Mayfield added the comment: Alexander, That sounds unrelated. I'd treat it as a new issue until you have concrete evidence to the contrary. Also on face value it sounds like it might just be your operating systems open file limit. On OSX I think the default open file limit is in the hundreds (256 on my box). Generally on unix-oid platforms it can be checked and changed with the `ulimit -n` command. Cheers -- ___ Python tracker <http://bugs.python.org/issue25593> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError
Justin Mayfield added the comment: I see. Seems like good discussion over there. I joined up. -- ___ Python tracker <http://bugs.python.org/issue25593> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError
Justin Mayfield added the comment: Ha, email race. Regarding rev 2, the updated docstring and scheduled stop looks good along with alleviating the confusion I mentioned. I'm not sure about your warning comment; Perhaps that's a patch I didn't lay eyes on. Cheers. -- ___ Python tracker <http://bugs.python.org/issue25593> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError
Justin Mayfield added the comment: I should have commented more on the run_once removal. The depiction given in its docstring seemed inconsistent with the new way stop works and I found no callers, so it seemed like it was best left out to avoid confusion. No worries though, I didn't get to know that test module very well before messing with it. It just came up in my scan for stop() callers. Looks good, I've applied to a 3.5.0 build and will include it in my testing from now on. Thanks Guido. -- ___ Python tracker <http://bugs.python.org/issue25593> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError
Justin Mayfield added the comment: Attached patch submission for stop flag proposal. I assume you didn't mean a github PR since the dev docs seem to indicate that is for readonly usage. This passes all the tests on my osx box but it should obviously be run by a lot more folks. -- Added file: http://bugs.python.org/file41059/Issue25593_fix.patch ___ Python tracker <http://bugs.python.org/issue25593> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError
Justin Mayfield added the comment: You bet. -- ___ Python tracker <http://bugs.python.org/issue25593> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError
Justin Mayfield added the comment: +1 Let me know what I can do to help. -- ___ Python tracker <http://bugs.python.org/issue25593> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError
Justin Mayfield added the comment: Yes, that's what I was suggesting. Looking at tornado they do the stop between callbacks/matured-scheduled and events. That approach seems somewhat arbitrary to me at first glance but tornado is very mature and they usually have good reasons for what they do. The notion of always completing a cycle seems more apt to me; Ie. your first design. A compelling thought experiment for allowing stop() to be lazy is if a user could somehow know when stop() was going to run or when it had been run. The nature of ioloop programming prevents you from knowing when it will run and because stop() has no return handle/future/task a user can't actually know when it did run. Ie. there is no way to await/add_done_callback on it, so baring hacks that bookend a stop() with marker callbacks it should be, as you said, sufficiently vague to justify a (more) lazy effect. -- I more or less agree on the s/cancelled/done/ changes. I'm using a similar monkey patch in my libraries to dance around this issue right now. I still don't exactly like the idea that code is written with an explicit expectation that it could be pending or cancelled, but then must also be inherently prepared for spurious done callbacks. This seems like a borderline contract violation by add_writer() and co. I suppose that add_writer() is primarily targeted at streams and the case of an EINTR in a socket connect() is a more a one-shot. Tough call. -- ___ Python tracker <http://bugs.python.org/issue25593> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError
Justin Mayfield added the comment: Interesting. I was going to do an analysis what using _ready.appendleft() for adding selector events would do for that scenario. The idea being to consistently juxtapose exiting callbacks, selector events and new callbacks. However I think this just moves the pawn in this ioloop halting problem. Is it worth investigating a change to the stop mechanism instead? Instead of raising an exception in the middle of run_once, it could set a flag to be seen by run_forever(). This may avoid this class of problem altogether and ensure run_once is a fairly simple and predictable. -- ___ Python tracker <http://bugs.python.org/issue25593> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError
Justin Mayfield added the comment: I don't believe this is a case of nonidempotent callbacks, unless you are referring to Future.set_result(), which by design can't be called twice. The callbacks are given an inconsistent opportunity to modify the poll set because of indeterminacy in the ioloop. That being said I understand your reluctance given the amount of turmoil this has but would argue that consistency with tornado is a powerful ally and that a model where any callback using call_soon will be guaranteed the opportunity to modify the poll set is a good thing. -- ___ Python tracker <http://bugs.python.org/issue25593> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError
Justin Mayfield added the comment: I'm attaching a patch that runs `_ready` callbacks at the start of `_run_once`. The style and implications are ranging so I leave it to you at this point. -- keywords: +patch Added file: http://bugs.python.org/file41019/run_once_testfix_for_Issue25593.patch ___ Python tracker <http://bugs.python.org/issue25593> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError
Justin Mayfield added the comment: Nevermind, in the case of writeablity it won't matter either way. -- So in looking at tornado's ioloop they run the ready callbacks before calling poll(). So the callbacks can modify the poll set. -- ___ Python tracker <http://bugs.python.org/issue25593> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError
Justin Mayfield added the comment: Guido, Shouldn't this not be the case for level triggered polling? From looking at selectors it looks like these are always level triggered which means they should only event once. -- ___ Python tracker <http://bugs.python.org/issue25593> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError
Justin Mayfield added the comment: Just reproduced on Linux, Fedora Core 23. -- ___ Python tracker <http://bugs.python.org/issue25593> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError
Justin Mayfield added the comment:
This code repros without aiohttp when pitted against the previously attached
web server (again on OSX 10.11, mid-2012 MBPr).
Admittedly this may seem very arbitrary but I have better reasons in my
production code for stopping an IOLoop and starting it again (which seems to be
important to the reproduction steps).
import asyncio
loop = asyncio.get_event_loop()
def batch_open():
for i in range(100):
c = asyncio.ensure_future(asyncio.open_connection('127.0.0.1', 8080))
c.add_done_callback(on_resp)
def on_resp(task):
task.result()
loop.stop()
loop.call_soon(batch_open)
while True:
loop.run_forever()
--
___
Python tracker
<http://bugs.python.org/issue25593>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError
Justin Mayfield added the comment: Attached server side of repro. -- Added file: http://bugs.python.org/file41017/Issue25593_repro_server.py ___ Python tracker <http://bugs.python.org/issue25593> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError
Justin Mayfield added the comment: Attaching simplified test setup. It does take some doing to repro so the local async server is required to make it happen (for me). When I tried just pointing to python.org it would not repro in 100 iterations, but using a local dummy server repros 100% for me. -- Added file: http://bugs.python.org/file41016/Issue25593_repro_client.py ___ Python tracker <http://bugs.python.org/issue25593> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError
Justin Mayfield added the comment: I believe I'm seeing this bug in a non-threaded and non-forked env. System: OSX 10.11.1 (15B42) Python 3.5.0 (from brew install) I'm using aiohttp to create several dozens of HTTP connections to the same server (an async tornado web server). Nothing special is being done around the event loop creation (standard get_event_loop()). However in my system the event loop is frequently stopped, via ioloop.stop(), and restarted via ioloop.run_forever(). I'm not sure this is related to the issue yet, but it's worth mentioning. I can't provide simplified test code just yet, but I can reproduce in my env with nearly 100% odds when doing a full system test. Attached is a sample backtrace. -- nosy: +Justin Mayfield Added file: http://bugs.python.org/file41015/asyncio_invalid_state_bt.txt ___ Python tracker <http://bugs.python.org/issue25593> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
