On 23May2019 17:04, bvdp <b...@mellowood.ca> wrote:
Anyway, yes the problem is that I was naively using command.getoutput()
which blocks until the command is finished. So, of course, only one process
was being run at one time! Bad me!
I guess I should be looking at subprocess.Popen(). Now, a more relevant
question ... if I do it this way I then need to poll though a list of saved
process IDs to see which have finished? Right? My initial thought is to
batch them up in small groups (say CPU_COUNT-1) and wait for that batch to
finish, etc. Would it be foolish to send send a large number (1200 in this
case since this is the number of files) and let the OS worry about
scheduling and have my program poll 1200 IDs?
Someone mentioned the GIL. If I launch separate processes then I don't
encounter this issue? Right?
Yes, but it becomes more painful to manage. If you're issues distinct
separate commands anyway, dispatch many or all and then wait for them as
a distinct step. If the commands start thrashing the rest of the OS
resources (such as the disc) then you may want to do some capacity
limitation, such as a counter or semaphore to limit how many go at once.
Now, waiting for a subcommand can be done in a few ways.
If you're then parent of all the processes you can keep a set() of the
issued process ids and then call os.wait() repeatedly, which returns the
pid of a completed child process. Check it against your set. If you need
to act on the specific process, use a dict to map pids to some record of
the subprocess.
Alternatively, you can spawn a Python Thread for each subcommand, have
the Thread dispatch the subcommand _and_ wait for it (i.e. keep your
command.getoutput() method, but in a Thread). Main programme waits for
the Threads by join()ing them.
Because a thread waiting for something external (the subprocess) doesn't
hold the GIL, other stuff can proceed. Basicly, if something is handed
off the to OS and then Python waits for that (via an os.* call or a
Popen.wait() call etc etc) then it will release the GIL while it is
blocked, so other Threads _will_ get to work.
This is all efficient, and there's any number of variations on the wait
step depending what your needs are.
The GIL isn't the disaster most people seem think. It can be a
bottleneck for pure Python compute intensive work. But Python's
interpreted - if you _really_ want performance the core compute will be
compiled to something more efficient (eg a C extension) or handed to
another process (transcode video in pure Python - argh! - but call the
ffmpeg command as a subprocess - yes!); handed off, the GIL should be
released, allowing other Python side work to continue.
Cheers,
Cameron Simpson <c...@cskk.id.au>
--
https://mail.python.org/mailman/listinfo/python-list
