On Sep 22, 10:44 am, Nobody <nob...@nowhere.com> wrote: > On Thu, 22 Sep 2011 08:55:40 -0700, Atherun wrote: > >> Just handle process.stdout/stderr by yourself - read it out until EOF > >> and then wait() for the process. > > > Thats what confuses me though, the documentation says > > process.stdout.read()/stderr.read() can deadlock and apparently so can > > communicate, how do you read the stdout/stderr on yourself if its > > documented using them can cause a deadlock? > > If you try to read/write two or more of stdin/stdout/stderr via the > "naive" approach, you run the risk of the child process writing more than > a pipe's worth of data to one stream (and thus blocking) while the > parent is performing a blocking read/write on another stream, resulting in > deadlock. > > The .communicate() method avoids the deadlock by either: > > 1. On Unix, using non-blocking I/O and select(), or > 2. On Windows, creating a separate thread for each stream. > > Either way, the result is that it can always read/write whichever > streams are ready, so the child will never block indefinitely while > waiting for the parent. > > If .communicate() is blocking indefinitely, it suggests that the child > process never terminates. There are many reasons why this might happen, > and most of them depend upon exactly what the child process is doing. > > I suggest obtaining a copy of Process Explorer, and using it to > investigate the state of both processes (but especially the child) at the > point that the "deadlock" seems to occur.
In the one case I can easily reproduce, its in a p4.exe call that I'm making both python and p4.exe have nearly the same stack for their threads: python: ntoskrnl.exe!memset+0x64a ntoskrnl.exe!KeWaitForMultipleObjects+0xd52 ntoskrnl.exe!KeWaitForMutexObject+0x19f ntoskrnl.exe!__misaligned_access+0xba4 ntoskrnl.exe!__misaligned_access+0x1821 ntoskrnl.exe!KeWaitForMultipleObjects+0xf5d ntoskrnl.exe!KeWaitForMutexObject+0x19f ntoskrnl.exe!NtWaitForSingleObject+0xde ntoskrnl.exe!KeSynchronizeExecution+0x3a43 wow64cpu.dll!TurboDispatchJumpAddressEnd+0x6c0 wow64cpu.dll!TurboDispatchJumpAddressEnd+0x4a8 wow64.dll!Wow64SystemServiceEx+0x1ce wow64.dll!Wow64LdrpInitialize+0x429 ntdll.dll!RtlUniform+0x6e6 ntdll.dll!RtlCreateTagHeap+0xa7 ntdll.dll!LdrInitializeThunk+0xe ntdll.dll!ZwWaitForSingleObject+0x15 kernel32.dll!WaitForSingleObjectEx+0x43 kernel32.dll!WaitForSingleObject+0x12 python26.dll!_Py_svnversion+0xcf8 p4: ntoskrnl.exe!memset+0x64a ntoskrnl.exe!KeWaitForMultipleObjects+0xd52 ntoskrnl.exe!KeWaitForSingleObject+0x19f ntoskrnl.exe!_misaligned_access+0xba4 ntoskrnl.exe!_misaligned_access+0x1821 ntoskrnl.exe!KeWaitForMultipleObjects+0xf5d ntoskrnl.exe!KeWaitForSingleObject+0x19f ntoskrnl.exe!NtCreateFile+0x4c9 ntoskrnl.exe!NtWriteFile+0x7e3 ntoskrnl.exe!KeSynchronizeExecution+0x3a43 ntdll.dll!ZwWriteFile+0xa KERNELBASE.dll!WriteFile+0x7b kernel32.dll!WriteFile+0x36 p4.exe+0x42d4b p4.exe+0x42ed8 To me it looks like they're both waiting on each other. -- http://mail.python.org/mailman/listinfo/python-list