On 5/9/2023 2:13 PM, Horst Koiner wrote:
Hi @all,
i'm running a program which is still in development with subprocess.run (Python
version 3.10), further i need to capture the output of the program in a python
variable. The program itself runs about 2 minutes, but it can also freeze in
case of new bugs.
For production i run the program with stdout=subprocess.PIPE and i can fetch
than the output later. For just testing if the program works, i run with
stdout=subprocess.STDOUT and I see all program output on the console, but my
program afterwards crashes since there is nothing captured in the python
variable. So I think I need to have the functionality of subprocess.PIPE and
subprcess.STDOUT sametime.
What I tried until now:
1. Poll the the output and use Popen instead:
# Start the subprocess
process = subprocess.Popen(['./test.sh'], stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
captured_output = b''
process_running = True
while process_running:
process_running = (process.poll() is not None)
for pipe in [ process.stdout, process.stderr ]:
while line := pipe.readline():
print(line)
captured_output += line
print(captured_output)
return_code = process.returncode
=> But this is discouraged by the python doc, since it says that polling this
way is prone to deadlocks. Instead it proposes the use of the communicate()
function.
2. Use communicate() with timeout.
=> This works not at all since when the timeout occurs an exception is thrown
and communicate returns at all.
3. Use threading instead
=> For being that simple and universal like subprocess you will more or less
reimplement subprocess with threading, like its done in subprocess.py. Just for a
debug output the effort is much to high.
#######################################################
Do you have further ideas for implementing such a behavior?
Do you think that a feature request should be done of I'm omitting something
obvious?
I'm not sure if this exactly fits your situation, but if you use
subprocess with pipes, you can often get a deadlock because the stdout
(or stderr, I suppose) pipe has a small capacity and fills up quickly
(at least on Windows), then it blocks until it is emptied by a read.
But if you aren't polling, you don't know there is something to read so
the pipe never gets emptied. And if you don't read it before the pipe
has filled up, you may lose data.
I solved that by running communicate() on a separate thread. Let the
communicate block the thread until the process has completed, then have
the thread send the result back to the main program. Of course, this
won't work if your process doesn't end since you won't get results until
the process ends.
--
https://mail.python.org/mailman/listinfo/python-list