[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError

Justin Mayfield added the comment: Alexander, That sounds unrelated. I'd treat it as a new issue until you have concrete evidence to the contrary. Also on face value it sounds like it might just be your operating systems open file limit. On OSX I think the default open file limit is in the

[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError

Changes by STINNER Victor : -- nosy: -haypo ___ Python tracker ___ ___

[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError

Alexander Mohr added the comment: update: its unrelated to the number of sessions or SSL, but instead to the number of concurrent aiohttp requests. When set to 500, I get the error, when set to 100 I do not. -- ___ Python tracker

[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError

Alexander Mohr added the comment: I'm not sure if you guys are still listening on this closed bug but I think I've found another issue ;) I'm using python 3.5.1 + asyncio 3.4.3 with the latest aiobotocore (which uses aiohttp 0.21.0) and had two sessions (two TCPConnectors), one doing a

[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError

Guido van Rossum added the comment: No, I mentioned the idea of a warning in the thread on the python-tulip mailing list, but decided not to do it after all. -- ___ Python tracker

[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError

Justin Mayfield added the comment: I should have commented more on the run_once removal. The depiction given in its docstring seemed inconsistent with the new way stop works and I found no callers, so it seemed like it was best left out to avoid confusion. No worries though, I didn't get to

[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError

Justin Mayfield added the comment: Ha, email race. Regarding rev 2, the updated docstring and scheduled stop looks good along with alleviating the confusion I mentioned. I'm not sure about your warning comment; Perhaps that's a patch I didn't lay eyes on. Cheers. --

[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError

Guido van Rossum added the comment: OK, here's another revision of the patch, setting the timeout passed to the selector to 0 when the loop is pre-stopped. -- Added file: http://bugs.python.org/file41084/issue25593_revised_3.patch ___ Python tracker

[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError

Guido van Rossum added the comment: OK, another revision, keep the mock selector. -- Added file: http://bugs.python.org/file41086/issue25593_revised_4.patch ___ Python tracker

[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError

Changes by Andrew Svetlov : -- nosy: +asvetlov ___ Python tracker ___ ___

[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError

Changes by Andrew Svetlov : -- versions: +Python 3.6 ___ Python tracker ___ ___

[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError

Roundup Robot added the comment: New changeset 9b3144716d17 by Guido van Rossum in branch '3.4': Issue #25593: Change semantics of EventLoop.stop(). https://hg.python.org/cpython/rev/9b3144716d17 New changeset 158cc5701488 by Guido van Rossum in branch '3.5': Issue #25593: Change semantics of

[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError

Guido van Rossum added the comment: Hopefully this is it! -- assignee: -> gvanrossum resolution: later -> fixed status: open -> closed ___ Python tracker

[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError

Guido van Rossum added the comment: New patch. Update test_utils.run_once() to use the recommended idiom. On second thought I don't like issuing a warning when stop() is called before the loop runs -- a warning seems overkill for something so minor. But I'm okay with no longer recommending

[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError

Justin Mayfield added the comment: I see. Seems like good discussion over there. I joined up. -- ___ Python tracker ___

[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError

Guido van Rossum added the comment: Whoops. Hopefully this one's right. -- Added file: http://bugs.python.org/file41087/issue25593_revised_5.patch ___ Python tracker

[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError

Guido van Rossum added the comment: I'm going to fix up the patch and apply it so this can make 3.5.1 rc1. -- ___ Python tracker ___

[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError

Guido van Rossum added the comment: Here's a better patch. - Renamed _stopped to _stopping. - Restore test_utils.run_once() and add a test for it. - Change logic so if _stopping is True upon entering run_forever(), it will run once. Please try it out!! --

[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError

Guido van Rossum added the comment: Here's the file. -- Added file: http://bugs.python.org/file41080/issue25593_revised.patch ___ Python tracker ___

[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError

Justin Mayfield added the comment: Attached patch submission for stop flag proposal. I assume you didn't mean a github PR since the dev docs seem to indicate that is for readonly usage. This passes all the tests on my osx box but it should obviously be run by a lot more folks. --

[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError

Guido van Rossum added the comment: Thinking about this more I believe it's possible for any of the FD callbacks in selector_events.py to be placed into loop._ready multiple times if the loop is stopped after the FD is ready (and the callback is scheduled) but before the callback is called.

[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError

Justin Mayfield added the comment: You bet. -- ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError

Guido van Rossum added the comment: @Justin: Do you want to come up with a PR for the stop() changes? Hopefully including tests (I bet at least one test will fail -- our tests in generally are pretty constraining). On Mon, Nov 16, 2015 at 6:32 PM, Justin Mayfield wrote:

[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError

Justin Mayfield added the comment: +1 Let me know what I can do to help. -- ___ Python tracker ___ ___

[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError

Alexander Mohr added the comment: btw want to thank you guys for actively looking into this, I'm very grateful! -- ___ Python tracker ___

[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError

Justin Mayfield added the comment: Interesting. I was going to do an analysis what using _ready.appendleft() for adding selector events would do for that scenario. The idea being to consistently juxtapose exiting callbacks, selector events and new callbacks. However I think this just moves

[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError

Guido van Rossum added the comment: Yeah, I've thought about changing the stop() mechanism too. It might mean that some callbacks will be executed that are currently skipped though, if your proposal is to run all callbacks in self._ready (excluding new ones) and then just exit if the stop flag

[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError

Guido van Rossum added the comment: I thought some more about this. The problem is due to stopping and restarting the loop, and that's also something that occurs in Alexander's example code, so I retract my accusation of aiohttp (and I don't think I need more investigation of his code). I

[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError

Justin Mayfield added the comment: Yes, that's what I was suggesting. Looking at tornado they do the stop between callbacks/matured-scheduled and events. That approach seems somewhat arbitrary to me at first glance but tornado is very mature and they usually have good reasons for what they

[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError

Alexander Mohr added the comment: Sorry for being obscure before, it was hard to pinpoint. I think I just figured it out! I had code like this in a subprocess: def worker(): while True: obj = self.queue.get() # do work with obj using asyncio http module def producer():

[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError

Alexander Mohr added the comment: clarification, adding the fut.done() check, or monkey patching: orig_sock_connect_cb = asyncio.selector_events.BaseSelectorEventLoop._sock_connect_cb def _sock_connect_cb(self, fut, sock, address): if fut.done(): return return orig_sock_connect_cb(self,

[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError

Alexander Mohr added the comment: Actually, I just realized I had fixed it locally by changing the callback to the following: 429 def _sock_connect_cb(self, fut, sock, address): 430 if fut.cancelled() or fut.done(): 431 return so a fix is still needed, and I also

[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError

Alexander Mohr added the comment: I'm going to close this as I've found a work-around, if I find a better test-case I'll open a new bug. -- resolution: -> later status: open -> closed ___ Python tracker

[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError

Guido van Rossum added the comment: I'm not an expert on this terminology but don't you have that backwards? Assume we're using select() for a second. If you ask select() "is this FD ready" several times in a row without doing something to the FD it will answer yes every time once the FD is

[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError

Justin Mayfield added the comment: I don't believe this is a case of nonidempotent callbacks, unless you are referring to Future.set_result(), which by design can't be called twice. The callbacks are given an inconsistent opportunity to modify the poll set because of indeterminacy in the

[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError

Justin Mayfield added the comment: Guido, Shouldn't this not be the case for level triggered polling? From looking at selectors it looks like these are always level triggered which means they should only event once. -- ___ Python tracker

[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError

Guido van Rossum added the comment: Thanks, but I don't like the idea of that patch. It feels like a hack that makes it less likely that the issue occurs, but I don't feel we should rely on the callbacks being called before checking the selector again. There may be other reasons (perhaps a

[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError

Justin Mayfield added the comment: Attached server side of repro. -- Added file: http://bugs.python.org/file41017/Issue25593_repro_server.py ___ Python tracker

[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError

Justin Mayfield added the comment: Just reproduced on Linux, Fedora Core 23. -- ___ Python tracker ___ ___

[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError

Alexander Mohr added the comment: attaching my simplified testcase and logged an aiohttp bug: https://github.com/KeepSafe/aiohttp/issues/633 -- Added file: http://bugs.python.org/file41018/test_app.py ___ Python tracker

[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError

Justin Mayfield added the comment: Attaching simplified test setup. It does take some doing to repro so the local async server is required to make it happen (for me). When I tried just pointing to python.org it would not repro in 100 iterations, but using a local dummy server repros 100%

[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError

Justin Mayfield added the comment: Nevermind, in the case of writeablity it won't matter either way. -- So in looking at tornado's ioloop they run the ready callbacks before calling poll(). So the callbacks can modify the poll set. -- ___ Python

[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError

Guido van Rossum added the comment: I wonder if the bug is in aiohttp? The code you show is still too complex to debug for me. -- ___ Python tracker ___

[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError

Justin Mayfield added the comment: This code repros without aiohttp when pitted against the previously attached web server (again on OSX 10.11, mid-2012 MBPr). Admittedly this may seem very arbitrary but I have better reasons in my production code for stopping an IOLoop and starting it again

[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError

Guido van Rossum added the comment: Justin's repro provides a clue: when the event loop is stopped before all callbacks have been processed, when the loop is restarted the I/O selector is asked again to do its work, and it will report all the same sockets as ready. So then the callback will be

[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError

Justin Mayfield added the comment: I'm attaching a patch that runs `_ready` callbacks at the start of `_run_once`. The style and implications are ranging so I leave it to you at this point. -- keywords: +patch Added file:

[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError

Guido van Rossum added the comment: Sorry, the code you posted is still incomprehensible. E.g. I suppose your worker doesn't really have ``` obj = self.queue.get() ``` but rather something like ``` obj = yield from async_queue.get() ``` But in the end, even with that hypothesis, I can't explain

[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError

Alexander Mohr added the comment: self.queue is not an async queue, as I stated above its a multiprocessing queue. This code is to multiplex a multiprocessing queue to a async queue. -- ___ Python tracker

[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError

Justin Mayfield added the comment: I believe I'm seeing this bug in a non-threaded and non-forked env. System: OSX 10.11.1 (15B42) Python 3.5.0 (from brew install) I'm using aiohttp to create several dozens of HTTP connections to the same server (an async tornado web server). Nothing

[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError

Alexander Mohr added the comment: Perhaps I'm doing something really stupid, but I was able to reproduce the two issues I'm having with the following sample script. If you leave the monkey patch disabled, you get the InvalidStateError, if you enable it, you get the ServerDisconnect errors

[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError

Guido van Rossum added the comment: Please show us how to repro -- there's no way we can figure out how this "impossible" event could happen in your code without understanding your code. Is it possible that multiprocessing forked your event loop or something similarly esoteric? --

[issue25593] _sock_connect_cb can be called twice resulting in InvalidStateError

New submission from Alexander Mohr: asyncio.selector_events.BaseSelectorEventLoop._sock_connect_cb is a callback based on the selector for a socket. There are certain situations when the selector triggers twice calling this callback twice, resulting in an InvalidStateError when it sets the