[issue1191964] asynchronous Subprocess

Changes by Giampaolo Rodola' : -- components: +Asyncio ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https:

[issue1191964] asynchronous Subprocess

akira added the comment: > First, with the use of Overlapped IO on Windows, BlockingIOError should never > come up, and we don't even handle that exception. Is it correct that the way to distinguish between "would block" and EOF is Popen.poll()? Is it possible to detect it in _nonblocking() me

[issue1191964] asynchronous Subprocess

Josiah Carlson added the comment: First, with the use of Overlapped IO on Windows, BlockingIOError should never come up, and we don't even handle that exception. As for BrokenPipeError vs. EOF; while we do treat them more or less the same, we aren't killing the process or reporting that the pr

[issue1191964] asynchronous Subprocess

akira added the comment: > Does anyone have questions, comments, or concerns about the patch? It seems the current API doesn't distinguish between BlockingIOError (temporary error), BrokenPipeError (permanent error) and EOF (permanent non-error condition) -- everything is treated as EOF. Is it

[issue1191964] asynchronous Subprocess

Guido van Rossum added the comment: FWIW while the Tulip changes should indeed go into the Tulip repo first, the rest should be committed to the CPython 3.5 tree first. Then I'll commit the Tulip changes, first to the Tulip repo, then to the CPython 3.4 branch (yes!) and then merge that into t

[issue1191964] asynchronous Subprocess

Josiah Carlson added the comment: I submitted an issue to the tulip/asyncio bug tracker: https://code.google.com/p/tulip/issues/detail?id=170 And I am uploading a new patch that only includes non-tulip/asyncio related changes, as tulip/asyncio changes will eventually be propagated to Python.

[issue1191964] asynchronous Subprocess

STINNER Victor added the comment: @Josiah: Modifications of the asyncio module should be done first in the Tulip project because we try to keep the same code base in Tulip, Python 3.4 and 3.5. You may duplicate the code the avoid this dependency? For the documentation, I don't think that you n

[issue1191964] asynchronous Subprocess

Josiah Carlson added the comment: First off, thank you everyone who has reviewed and commented so far. I very much appreciate your input and efforts. Does anyone have questions, comments, or concerns about the patch? If no one mentions anything in the next week or so, I'll ping the email threa

[issue1191964] asynchronous Subprocess

Josiah Carlson added the comment: Richard: short timeouts are no longer an issue. Nothing to worry about :) -- ___ Python tracker ___ __

[issue1191964] asynchronous Subprocess

Richard Oudkerk added the comment: If you use the short timeouts to make the wait interruptible then you can use waitformultipleobjects (which automatically waits on an extra event object) instead of waitforsingleobject. -- ___ Python tracker

[issue1191964] asynchronous Subprocess

Josiah Carlson added the comment: Victor, I addressed the majority of your comments except for a couple stylistic pieces. Your annoyance with the short poll time for Windows made me re-read the docs from MS, which made me realize that my interpretation was wrong. It also made me confirm Richar

[issue1191964] asynchronous Subprocess

Changes by Martin Panter : -- nosy: +vadmium ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.pyt

[issue1191964] asynchronous Subprocess

STINNER Victor added the comment: I started to review the patch 5: http://bugs.python.org/review/1191964/#ps11598 When I read unit tests, I realized that I don't like "write_nonblocking" name. It's too generic. A process has many files (more than just stdin, stdout, stderr: see pass_fds parame

[issue1191964] asynchronous Subprocess

STINNER Victor added the comment: I suggest to change the title of the issue to: "subprocess: add non-blocking read and write methods" to avoid the confusion with asyncio subprocess module which runs read and write "in the background" for you. -- nosy: +haypo _

[issue1191964] asynchronous Subprocess

Changes by Josiah Carlson : Added file: http://bugs.python.org/file34861/subprocess_5.patch ___ Python tracker ___ ___ Python-bugs-list mail

[issue1191964] asynchronous Subprocess

Josiah Carlson added the comment: I ended up eliminating the overlapped IO cancel call on Windows. Better to be correct than to minimize internal state. Instead, we keep a reference to the overlapped IO object, and any attempts to write to the child stdin before the previous overlapped IO comp

[issue1191964] asynchronous Subprocess

Josiah Carlson added the comment: No, the problem is that that ov.cancel() will attempt to cancel the IO, but a subsequent ov.getresult(True) doesn't always return what was *actually* written to the pipe unless you explicitly wait for the result to be available. But even if you explicitly wait

[issue1191964] asynchronous Subprocess

Richard Oudkerk added the comment: I added some comments. Your problem with lost data may be caused by the fact you call ov.cancel() and expect ov.pending to tell you whether the write has/will succeed. Instead you should use ov.getresult() and expect either success or an "aborted" error. --

[issue1191964] asynchronous Subprocess

Josiah Carlson added the comment: I added the chunking for Windows because in manual testing before finishing the patch, I found that large sends on Windows without actually waiting for the result can periodically result in zero data sent, despite a child process that wants to read. Looking a

[issue1191964] asynchronous Subprocess

Richard Oudkerk added the comment: Can you explain why you write in 512 byte chunks. Writing in one chunk should not cause a deadlock. -- ___ Python tracker ___ _

[issue1191964] asynchronous Subprocess

Josiah Carlson added the comment: Submitting an updated patch addressing Giampaolo's comments. -- Added file: http://bugs.python.org/file34774/subprocess_2.patch ___ Python tracker ___

[issue1191964] asynchronous Subprocess

akira added the comment: Could `read_nonblocking()`, `write_nonblocking()` raise OSError(EAGAIN) (it could be named `ResourceTemporarilyUnavailableError`) if read/write would block? It would allow to distinguish EOF (permanent condition) from "read/write would block" (temporarily condition) wi

[issue1191964] asynchronous Subprocess

akira added the comment: > Also, Richard Oudkerk's claims above about not needing to use fcntl to swap > flags is not correct. It's necessary to not block on reading, even if select > is used. Select just guarantees that there is at least 1 byte or a closed > handle, not that your full read wi

[issue1191964] asynchronous Subprocess

Giampaolo Rodola' added the comment: Some initial comments here: http://bugs.python.org/review/1191964/#msg1. Also, if I think about integrating this into an IO loop it seems natural to me to think about timeouts. How complicated would it be to do this? data = proc.read_nonblocking(timeout=0) d

[issue1191964] asynchronous Subprocess

Josiah Carlson added the comment: Should have uploaded this yesterday, but I got caught up with typical weekend activities. The docs probably need more, and I hear that select.select() is disliked, so that will probably need to be changed too. -- Added file: http://bugs.python.org/file

[issue1191964] asynchronous Subprocess

Josiah Carlson added the comment: All of the standard tests plus another few that I added all pass on Windows and Linux. I've got some cleanup and a couple more tests to add tomorrow, then I'll post a patch. I ended up not using any overlapped IO cancellation in the Windows variant of communi

[issue1191964] asynchronous Subprocess

Josiah Carlson added the comment: Quick update before I head to bed. Thank you for the input, I had gotten the individual async calls working a couple days ago, and I was just working to replace the communicate() method for Windows. Yes, I'm using asyncio._overlapped, though asyncio uses subp

[issue1191964] asynchronous Subprocess

Richard Oudkerk added the comment: I would recommended using _overlapped instead of _winapi. I intend to move multiprocessing over in future. Also note that you can do nonblocking reads by starting an overlapped read then cancelling it immediately if it fails with "incomplete". You will need to

[issue1191964] asynchronous Subprocess

eryksun added the comment: multiprocessing.connection uses the _winapi module for overlapped I/O and named pipes in case you're looking for examples: http://hg.python.org/cpython/file/3.4/Lib/multiprocessing/connection.py -- nosy: +eryksun ___ Pytho

[issue1191964] asynchronous Subprocess

Josiah Carlson added the comment: Had some time to work on this today. I was missing something in my earlier versions of the code, and have managed to get overlapped IOs to work, so at least I'm not quite so far behind the dozen or so core developers who know more about the Windows pieces than

[issue1191964] asynchronous Subprocess

Changes by Glenn Linderman : -- nosy: +v+python ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.

[issue1191964] asynchronous Subprocess

Changes by janzert : -- nosy: +janzert ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.or

[issue1191964] asynchronous Subprocess

Richard Oudkerk added the comment: Using asyncio and the IOCP eventloop it is not necessary to use threads. (Windows may use worker threads for overlapped IO, but that is hidden from Python.) See https://code.google.com/p/tulip/source/browse/examples/child_process.py for vaguely "expect-l

[issue1191964] asynchronous Subprocess

Changes by Josiah Carlson : -- versions: +Python 3.5 -Python 3.3, Python 3.4 ___ Python tracker ___ ___ Python-bugs-list mailing lis

[issue1191964] asynchronous Subprocess

Josiah Carlson added the comment: Due to some rumblings over on the mentors list and python-dev, this is now getting some love. Guido has stated that something should make it into the subprocess module for 3.5 in this email: https://groups.google.com/d/msg/dev-python/I6adJLIjNHk/Usrvxe_PVJIJ

[issue1191964] asynchronous Subprocess

Changes by Mark Lawrence : -- nosy: -BreamoreBoy ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mai

[issue1191964] asynchronous Subprocess

anatoly techtonik added the comment: Can anybody write a sum up on this? -- versions: +Python 3.4 ___ Python tracker ___ ___ Python-

[issue1191964] asynchronous Subprocess

Richard Oudkerk added the comment: > How would this differ from the normal communicate()? It would block until one of the following occurs: * some data has been written to stdin, * some data has been read from stdout or stderr, or * timeout passes (if timeout is not None). The normal communic

[issue1191964] asynchronous Subprocess

Ross Lagerwall added the comment: > Personally, I would factor out the code for Popen.communicate() in to a > > Communicator class which wraps a Popen object and has a method > >communicate(input, timeout=None) -> (bytes_written, output, error) How would this differ from the normal communi

[issue1191964] asynchronous Subprocess

Changes by Chris Rebert : -- nosy: +cvrebert ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.pyth

[issue1191964] asynchronous Subprocess

Eric Pruitt added the comment: There's documentation, but you have to switch to the Python 3k branch -- http://code.google.com/p/subprocdev/source/browse/?name=python3k#hg%2Fdoc. As for the other criticisms, I'm sure there are plenty of things that need to be improved upon; I was not a very exp

[issue1191964] asynchronous Subprocess

Richard Oudkerk added the comment: Personally, I would factor out the code for Popen.communicate() in to a Communicator class which wraps a Popen object and has a method communicate(input, timeout=None) -> (bytes_written, output, error) On Windows this would use threads, and on Unix, sele

[issue1191964] asynchronous Subprocess

Richard Oudkerk added the comment: Comments on Josiah's patch: * It uses pywin32 for PeekNamedPipe -- this is now available from _winapi. * I don't think send(), recv() and recv_exact() will work correctly if buffering is used -- an error should be raised in this case. * I think send(), rec

[issue1191964] asynchronous Subprocess

Changes by Ross Lagerwall : -- nosy: +rosslagerwall ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://ma

[issue1191964] asynchronous Subprocess

Changes by Ross Lagerwall : -- assignee: astrand -> ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://m

[issue1191964] asynchronous Subprocess

anatoly techtonik added the comment: What is the status? What is required to get this into 3.3? See also issue14872. -- nosy: +techtonik ___ Python tracker ___ ___

[issue1191964] asynchronous Subprocess

Changes by Giampaolo Rodola' : -- versions: +Python 3.3 -Python 3.2 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubsc

[issue1191964] asynchronous Subprocess

Changes by Andrew Boettcher : -- nosy: +Andrew.Boettcher ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http

[issue1191964] asynchronous Subprocess

Mark Lawrence added the comment: PEP 3145 has been written in response to this request. -- nosy: +BreamoreBoy versions: -Python 2.7 ___ Python tracker ___ ___

[issue1191964] asynchronous Subprocess

R. David Murray added the comment: Retargeting to 3.2 since 3.1 is now feature-frozen. -- nosy: +r.david.murray versions: +Python 3.2 -Python 3.1 ___ Python tracker ___ __

[issue1191964] asynchronous Subprocess

James Eric Pruitt added the comment: Hello, I am working on this patch and some additional features for my Google Summer of Code project (subdev.blogspot.com) and will eventually attempt to get the code committed to Python 3.1 or 3.2 and Python 2.7. I will have the unit tests completed shortly a

[issue1191964] asynchronous Subprocess

Josiah Carlson added the comment: I don't believe this should be closed. The functionality is still desired by me and others who have posted on and off since the patch was created. This patch definitely needs some love (tests mostly). -- ___ Pyth

[issue1191964] asynchronous Subprocess

Changes by Giampaolo Rodola' : -- nosy: +giampaolo.rodola ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: htt

[issue1191964] asynchronous Subprocess

Daniel Diniz added the comment: Josiah: can this be closed? -- components: +Library (Lib) -None keywords: +patch nosy: +ajaksu2 stage: -> test needed versions: +Python 2.7, Python 3.1 ___ Python tracker ___