[issue19293] test_asyncio hanging for 1 hour (AIX version, hangs in test_subprocess_interactive)

David Edelsohn added the comment: You have to try harder to break it again. aixfix2.diff still works. ;-) -- ___ Python tracker ___ __

[issue19293] test_asyncio hanging for 1 hour (AIX version, hangs in test_subprocess_interactive)

Guido van Rossum added the comment: "And there was much rejoicing." -- resolution: -> fixed stage: -> committed/rejected status: open -> closed ___ Python tracker ___ _

[issue19293] test_asyncio hanging for 1 hour (AIX version, hangs in test_subprocess_interactive)

Roundup Robot added the comment: New changeset c2e018c54689 by Guido van Rossum in branch 'default': Fix asyncio issue #19293 (hangs on AIX). http://hg.python.org/cpython/rev/c2e018c54689 -- ___ Python tracker

[issue19293] test_asyncio hanging for 1 hour (AIX version, hangs in test_subprocess_interactive)

Changes by Guido van Rossum : Removed file: http://bugs.python.org/file32291/aixfix.diff ___ Python tracker ___ ___ Python-bugs-list mailing l

[issue19293] test_asyncio hanging for 1 hour (AIX version, hangs in test_subprocess_interactive)

Changes by Guido van Rossum : -- Removed message: http://bugs.python.org/msg200878 ___ Python tracker ___ ___ Python-bugs-list mailing

[issue19293] test_asyncio hanging for 1 hour (AIX version, hangs in test_subprocess_interactive)

Guido van Rossum added the comment: Third try's a charm. Corrected aix fix. -- Added file: http://bugs.python.org/file32292/aixfix2.diff ___ Python tracker ___ __

[issue19293] test_asyncio hanging for 1 hour (AIX version, hangs in test_subprocess_interactive)

Guido van Rossum added the comment: Corrected aixfix.diff for CPython repo (the previous one was for the Tulip repo). -- ___ Python tracker ___ _

[issue19293] test_asyncio hanging for 1 hour (AIX version, hangs in test_subprocess_interactive)

Changes by Guido van Rossum : -- Removed message: http://bugs.python.org/msg200877 ___ Python tracker ___ ___ Python-bugs-list mailing

[issue19293] test_asyncio hanging for 1 hour (AIX version, hangs in test_subprocess_interactive)

David Edelsohn added the comment: test_asyncio no longer hangs on AIX with the aixfix.diff patch. Thanks! -- ___ Python tracker ___ __

[issue19293] test_asyncio hanging for 1 hour (AIX version, hangs in test_subprocess_interactive)

Guido van Rossum added the comment: Corrected aixfix.diff (the other one was for the Tulip repo). -- ___ Python tracker ___ ___ Python

[issue19293] test_asyncio hanging for 1 hour (AIX version, hangs in test_subprocess_interactive)

Guido van Rossum added the comment: David, try this fix (aixfix.diff). -- Added file: http://bugs.python.org/file32291/aixfix.diff ___ Python tracker ___

[issue19293] test_asyncio hanging for 1 hour (AIX version, hangs in test_subprocess_interactive)

Guido van Rossum added the comment: I think I know what that is -- the test should use a socketpair now. I'll see how quickly I can give you a patch for that. The other one is now committed. -- ___ Python tracker

[issue19293] test_asyncio hanging for 1 hour (AIX version, hangs in test_subprocess_interactive)

David Edelsohn added the comment: With the latest changeset applied to default, the test_subprocess tests no longer hang, but test_write_pipe_disconnect_on_close now hangs. The attached patch changes test_event.py to not skip test_subprocess_* but now skips test_write_pipe_disconnect_on_close.

[issue19293] test_asyncio hanging for 1 hour (AIX version, hangs in test_subprocess_interactive)

Guido van Rossum added the comment: Let's see how this fares. If it passes on the AIX buildbot, maybe see which of the tests we skipped on AIX can be re-enabled again? -- ___ Python tracker ___

[issue19293] test_asyncio hanging for 1 hour (AIX version, hangs in test_subprocess_interactive)

Roundup Robot added the comment: New changeset 2a0bda8d283d by Guido van Rossum in branch 'default': Switch subprocess stdin to a socketpair, attempting to fix issue #19293 (AIX hang). http://hg.python.org/cpython/rev/2a0bda8d283d -- ___ Python track

[issue19293] test_asyncio hanging for 1 hour (AIX version, hangs in test_subprocess_interactive)

Guido van Rossum added the comment: Thanks; I've applied and pushed your fix for the hang. I think it's time to commit the sockpair.diff change; the test_write_pipe() failure looks related -- as you recall the problem doesn't really cause a hang, it causes a premature closing of a write pipe,

[issue19293] test_asyncio hanging for 1 hour (AIX version, hangs in test_subprocess_interactive)

David Edelsohn added the comment: Other than the skipped tests, test_asyncio now passes on AIX except for: FAIL: test_write_pipe (test.test_asyncio.test_events.PollEventLoopTests) -- Traceback (most recent call last): File "/h

[issue19293] test_asyncio hanging for 1 hour (AIX version, hangs in test_subprocess_interactive)

David Edelsohn added the comment: The testsuite is hanging in test_asyncio again and test_subprocess_close_client_stream needs to be skipped. -- Added file: http://bugs.python.org/file32289/issue19293.patch2 ___ Python tracker

[issue19293] test_asyncio hanging for 1 hour (AIX version, hangs in test_subprocess_interactive)

Guido van Rossum added the comment: New, improved version of sockpair.diff. -- Added file: http://bugs.python.org/file32284/sockpair.diff ___ Python tracker ___ _

[issue19293] test_asyncio hanging for 1 hour (AIX version, hangs in test_subprocess_interactive)

Changes by Guido van Rossum : Removed file: http://bugs.python.org/file32282/sockpair.diff ___ Python tracker ___ ___ Python-bugs-list mailing

[issue19293] test_asyncio hanging for 1 hour (AIX version, hangs in test_subprocess_interactive)

Guido van Rossum added the comment: Attached is a version of Richard's patch that I like better (sockpair.diff). - instead of dropping the ValueError, check for ISFIFO and ISSOCK - pass the read end of the socket pair to Popen() -- Added file: http://bugs.python.org/file32282/sockpair.d

[issue19293] test_asyncio hanging for 1 hour (AIX version, hangs in test_subprocess_interactive)

Guido van Rossum added the comment: Brilliant. It works for me too on OS X. David, can you check this on AIX? -- ___ Python tracker ___ _

[issue19293] test_asyncio hanging for 1 hour (AIX version, hangs in test_subprocess_interactive)

Richard Oudkerk added the comment: The following uses socketpair() instead of pipe() for stdin, and works for me on Linux: diff -r 7d94e4a68b91 asyncio/unix_events.py --- a/asyncio/unix_events.pySun Oct 20 20:25:04 2013 -0700 +++ b/asyncio/unix_events.pyMon Oct 21 17:15:19 2013 +0100 @@

[issue19293] test_asyncio hanging for 1 hour (AIX version, hangs in test_subprocess_interactive)

Changes by Florent Xicluna : -- nosy: +flox ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python

[issue19293] test_asyncio hanging for 1 hour (AIX version, hangs in test_subprocess_interactive)

Guido van Rossum added the comment: @Richard: socketpair() seems to work fine on AIX (we use it internally). Maybe we can always use that for subprocess management? But will the subprocess module support this? -- ___ Python tracker

[issue19293] test_asyncio hanging for 1 hour (AIX version, hangs in test_subprocess_interactive)

Guido van Rossum added the comment: OK, I've created issue 19334 to track test_asyncio hangs on other platforms. Let's please keep this issue restricted to AIX. I am aware of the following issue: - select for read on the write end of a pipe returns immediately Any other AIX issues also please

[issue19293] test_asyncio hanging for 1 hour

Richard Oudkerk added the comment: Would it make sense to use socketpair() instead of pipe() on AIX? We could check for the "bug" directly rather than checking specifically for AIX. -- ___ Python tracker

[issue19293] test_asyncio hanging for 1 hour

Charles-François Natali added the comment: I'm completely lost: how many issues is this thread conflating? For example, David, I think you said that my patch regarding signal handling did fix some tests. Is that still the case? If yes, could you open a specific issue for the remaining failing tes

[issue19293] test_asyncio hanging for 1 hour

STINNER Victor added the comment: 2013/10/21 David Edelsohn : >> I added many tests when I added this feature to Python: see test_signal.py. >> By the way, it uses select.select() :-) Does test_signal pass successfully >> on AIX? > > @haypo: test_signal skips some tests on AIX. See Issue #1823

[issue19293] test_asyncio hanging for 1 hour

David Edelsohn added the comment: > I added many tests when I added this feature to Python: see test_signal.py. > By the way, it uses select.select() :-) Does test_signal pass successfully on > AIX? @haypo: test_signal skips some tests on AIX. See Issue #18238 . -- _

[issue19293] test_asyncio hanging for 1 hour

Tim Peters added the comment: Just wondering: why is the timeout-to-failure set to a whole hour? Based on all the messages I've seen this weekend, we could save the buildbots countless years of time by reducing it to - say - 59 minutes ;-) -- nosy: +tim.peters __

[issue19293] test_asyncio hanging for 1 hour

STINNER Victor added the comment: > Hm... Maybe the wakeup_fd machinery is broken on some systems? Hey! I added many tests when I added this feature to Python: see test_signal.py. By the way, it uses select.select() :-) Does test_signal pass successfully on AIX? -- __

[issue19293] test_asyncio hanging for 1 hour

STINNER Victor added the comment: [Off-topic: Heck, another one of those upside-down tracebacks. WHY???] faulthandler is really a very low-level module. I wrote it to analyze deadlocks, memory corruptions and other sad cases. faulthandler is designed to provide as much information as possible

[issue19293] test_asyncio hanging for 1 hour

Guido van Rossum added the comment: No, there's a use case for reading after the child exited, if there is a grandchild still writing. --Guido van Rossum (sent from Android phone) On Oct 20, 2013 10:37 AM, "Richard Oudkerk" wrote: > > Richard Oudkerk added the comment: > > > I guess we'll have

[issue19293] test_asyncio hanging for 1 hour

Richard Oudkerk added the comment: > I guess we'll have to write platform-dependent code and make this an > optional feature. (Essentially, on platforms like AIX, for a > write-pipe, connection_lost() won't be called unless you try to write > some more bytes to it.) If we are not capturing stdou

[issue19293] test_asyncio hanging for 1 hour

Charles-François Natali added the comment: > I guess we'll have to write platform-dependent code and make this an > optional feature. (Essentially, on platforms like AIX, for a > write-pipe, connection_lost() won't be called unless you try to write > some more bytes to it.) > > I do believe that

[issue19293] test_asyncio hanging for 1 hour

Guido van Rossum added the comment: > > Could it be that selecting for *read* on the *write* end of a pipe is > > always ready? > > That's exactly what I was thinking when I read the code below: that's > definitely a possibility on AIX. David confirmed that it is the _read_ready() that closes th

[issue19293] test_asyncio hanging for 1 hour

David Edelsohn added the comment: For completeness, the highlights of the new truss trace output after the echo.py change and only tracing the main process to avoid confusion from the interleaved output: test_subprocess_interactive (test.test_asyncio.test_events.PollEventLoopTests) ... socket

[issue19293] test_asyncio hanging for 1 hour

David Edelsohn added the comment: > To test this theory, it should be sufficient to comment out > self._loop.add_reader(self._fileno, self._read_ready) When I comment out this line, test_subprocess_interactive succeeds on AIX. -- ___ Python tracker

[issue19293] test_asyncio hanging for 1 hour

Charles-François Natali added the comment: > Guido van Rossum added the comment: > >> Apparently, the stdout pipe was closed by the parent process > > Could it be that selecting for *read* on the *write* end of a pipe is > always ready? That's exactly what I was thinking when I read the code bel

[issue19293] test_asyncio hanging for 1 hour

Guido van Rossum added the comment: > Apparently, the stdout pipe was closed by the parent process Could it be that selecting for *read* on the *write* end of a pipe is always ready? In _UnixWritePipeTransport there's a read handler that immediately closes the pipe as soon as it called. I vaguel

[issue19293] test_asyncio hanging for 1 hour

Charles-François Natali added the comment: > David Edelsohn added the comment: > > AIX has an equivalent to strace (called truss). I have recorded all AIX > system calls and signals for test_process_interactive, which hangs, following > all children created by fock. The uncompressed file is 82M

[issue19293] test_asyncio hanging for 1 hour

Guido van Rossum added the comment: I'm trying to let go of the AIX hang. Here's a brain dump of what I've figured out so far. * There were a lot of red herrings in the early discussion. This hang doesn't seem to have anything to do with nonblocking connect() or sockets, nor even signals. *

[issue19293] test_asyncio hanging for 1 hour

Guido van Rossum added the comment: Let's just release the alpha with this as it stands; I will look at the race condition later, and also at CF's patch (which may be a good idea regardless of what it fixes). I did add the breaks to echo and echo3 since the busy-waiting just makes tracing thi

[issue19293] test_asyncio hanging for 1 hour

David Bolen added the comment: I've been trying to test this on the Ubuntu 8.04 buildbot, but so far, using the latest hg default head, haven't gotten test_asyncio to hang even before the patch. It also looks like the last 4 regular buildbot builds were fine too. I even tried rolling back to

[issue19293] test_asyncio hanging for 1 hour

Guido van Rossum added the comment: I wonder if subprocess pipes have a version of the race condition I fixed for sockets yesterday: http://code.google.com/p/tulip/source/detail?r=2d392599fa067038b5fc9eeaf051011307aeadee The pipe code seems derived from the socket code long ago. -- __

[issue19293] test_asyncio hanging for 1 hour

David Edelsohn added the comment: I added if not buf: break after buf = os.read() to echo.py, echo2.py and echo3.py. They now exit and do not spin, but the test still hangs. -- ___ Python tracker

[issue19293] test_asyncio hanging for 1 hour

Antoine Pitrou added the comment: > test_subprocess_interactive > (test.test_asyncio.test_events.PollEventLoopTests) ... > 4588380:64422135: close(3) = 0 > 4588380:64422135: > statx("/home/dje/src/cpython/Lib/test/test_asyncio/echo.py", 0x2FF20870,

[issue19293] test_asyncio hanging for 1 hour

David Edelsohn added the comment: Separate question for David: if CF's patch were committed, would we still need your patch to disable three subprocess-related tests on AIX? (That can be dealt with after the alpha release though -- in fact I propose not to wait for AIX if we can get the other

[issue19293] test_asyncio hanging for 1 hour

David Edelsohn added the comment: AIX has an equivalent to strace (called truss). I have recorded all AIX system calls and signals for test_process_interactive, which hangs, following all children created by fock. The uncompressed file is 82MB or 939KB compressed. The highlights are: test_su

[issue19293] test_asyncio hanging for 1 hour

Antoine Pitrou added the comment: RHEL 6 too: http://buildbot.python.org/all/builders/x86%20RHEL%206%203.x/builds/2911/steps/test/logs/stdio -- nosy: +ncoghlan ___ Python tracker ___

[issue19293] test_asyncio hanging for 1 hour

Guido van Rossum added the comment: Separate question for David: if CF's patch were committed, would we still need your patch to disable three subprocess-related tests on AIX? (That can be dealt with after the alpha release though -- in fact I propose not to wait for AIX if we can get the othe

[issue19293] test_asyncio hanging for 1 hour

Antoine Pitrou added the comment: I don't have access to an Ubuntu 8.04 box, but if David Bolen can test the patch it would be great. -- nosy: +db3l ___ Python tracker ___ _

[issue19293] test_asyncio hanging for 1 hour

Guido van Rossum added the comment: Hm... Maybe the wakeup_fd machinery is broken on some systems? Has anybody been able to confirm that CF's patch fixes the hang on any non-AIX system? If so, maybe we should commit first, understand later... -- ___

[issue19293] test_asyncio hanging for 1 hour

Charles-François Natali added the comment: > Guido van Rossum added the comment: > > Actually, I think we're talking about the actual select() syscal not > returning when SIGCHLD is received and handled. Running the C-level handler > isn't enough, the syscall must return with EINTR so the Python

[issue19293] test_asyncio hanging for 1 hour

Guido van Rossum added the comment: If CF's patch fixes the Ubuntu hang, commit it! We can iterate on my nits post-alpha. -- ___ Python tracker ___

[issue19293] test_asyncio hanging for 1 hour

Larry Hastings added the comment: There's also a test that hangs for 1 hour on the Ubuntu x86 buildbot. Sample failure: http://buildbot.python.org/all/builders/x86%20Ubuntu%20Shared%203.x/builds/8803/ -- nosy: +larry ___ Python tracker

[issue19293] test_asyncio hanging for 1 hour

Antoine Pitrou added the comment: The exact same hanging seems to happen under Ubuntu 8.04 LTS: http://buildbot.python.org/all/builders/x86%20Ubuntu%20Shared%203.x/builds/8803/steps/test/logs/stdio [176/383] test_asyncio Timeout (1:00:00)! Thread 0x404e48c0: File "/srv/buildbot/buildarea/3.x.

[issue19293] test_asyncio hanging for 1 hour

Guido van Rossum added the comment: I guess it's a shame that the two issues got merged, since now it looks like they may have to be unmerged again. I sent a few comments to neologix' patch that should be easy to address; for AIX I'm afraid we're still looking for more info from David. --

[issue19293] test_asyncio hanging for 1 hour

Charles-François Natali added the comment: > The patch cleans up a lot of the other failures for asyncio on AIX Great. > but does not fix the hang. I was expecting this :-) Note that the hang is really strange, especially since the first run_until_complete() succeeds. Without more informatio

[issue19293] test_asyncio hanging for 1 hour

David Edelsohn added the comment: The patch cleans up a lot of the other failures for asyncio on AIX, but does not fix the hang. -- ___ Python tracker ___ __

[issue19293] test_asyncio hanging for 1 hour

Charles-François Natali added the comment: Here's a patch changing asyncio to not rely on a Python-level signal handler at all: it reads the signal number from the self-pipe. It should therefore work even if the select()/poll()/etc syscall doesn't fail with EINTR, and incidentally require only

[issue19293] test_asyncio hanging for 1 hour

Antoine Pitrou added the comment: > [Off-topic: Heck, another one of those upside-down tracebacks. WHY???] These are tracebacks emitted by the faulthandler module. They don't use the normal traceback printing machinery because it is impossible to do so in the conditions where those tracebacks

[issue19293] test_asyncio hanging for 1 hour

Charles-François Natali added the comment: > Guido van Rossum added the comment: > > Actually, I think we're talking about the actual select() syscal not > returning when SIGCHLD is received and handled. Running the C-level handler > isn't enough, the syscall must return with EINTR so the Python

[issue19293] test_asyncio hanging for 1 hour

David Edelsohn added the comment: poll() vs select() does not make a difference. Using the default poll(), if I run python in GDB, test_process_interactive hangs in poll(). The top of the traceback looks like: #0 0xd02389d4 in __fd_poll () from /usr/lib/libc.a(shr.o) #1 0xd717e604 in poll (_

[issue19293] test_asyncio hanging for 1 hour

Guido van Rossum added the comment: Actually, I think we're talking about the actual select() syscal not returning when SIGCHLD is received and handled. Running the C-level handler isn't enough, the syscall must return with EINTR so the Python handler can run (which then writes a byte to the self

[issue19293] test_asyncio hanging for 1 hour

David Edelsohn added the comment: Check that which processes are terminated? The test eventually timed out after 1 hour and the process running echo.py continued to run. -- ___ Python tracker _

[issue19293] test_asyncio hanging for 1 hour

Charles-François Natali added the comment: Wait, I think there's a misunderstanding here. POSIX does indeed leave undefined the behavior of syscalls when a signal is received: whether or not it fails with EINTR depends on the operating system, the SA_RESTART flag, etc. But the tests we're talkin

[issue19293] test_asyncio hanging for 1 hour

David Edelsohn added the comment: Also, the Python buildbot for AIX is running on a POWER AIX 7.1 system at OSU OSL that is a member of the "GCC Compile Farm" cluster on which FOSS developers can obtain free accounts. If you want access to an AIX system for testing, it's available. --

[issue19293] test_asyncio hanging for 1 hour

David Edelsohn added the comment: My understanding of the issue is POSIX and The Open Group specification are ambiguous about the handling of Unix socket API calls when interrupted by a signal. Linux implements a more liberal, accommodating behavior and the Python Async I/O implementation is a

[issue19293] test_asyncio hanging for 1 hour

Changes by Guido van Rossum : -- assignee: -> gvanrossum ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https

[issue19293] test_asyncio hanging for 1 hour

Guido van Rossum added the comment: [Off-topic: Heck, another one of those upside-down tracebacks. WHY???] I'll need a lot of help from folks on the affected platforms in debugging this, since these tests work fine when I run them on the platforms to which I have access (Ubuntu and OS X). I w

[issue19293] test_asyncio hanging for 1 hour

Changes by Stefan Krah : -- title: test_asyncio failures on AIX -> test_asyncio hanging for 1 hour ___ Python tracker ___ ___ Python-b