Tim Arnold wrote:
"Jan Kaliszewski" <z...@chopin.edu.pl> wrote in message news:mailman.895.1251958800.2854.python-l...@python.org...
06:49:13 Scott David Daniels <scott.dani...@acm.org> wrote:

Tim Arnold wrote:
(1) what's wrong with having each chapter in a separate thread? Too much going on for a single processor?
Many more threads than cores and you spend a lot of your CPU switching
tasks.
In fact, python threads work relatively the best with a powerful single
core; with more cores it becomes being suprisingly inefficient.

The culprit is Pythn GIL and the way it [mis]cooperates with OS
scheduling.

See: http://www.dabeaz.com/python/GIL.pdf

Yo
*j

--
Jan Kaliszewski (zuo) <z...@chopin.edu.pl>

I've read about the GIL (I think I understand the problem there)--thanks. In my example, the actual job called for each chapter ended up being a call to subprocess (that called a different python program). I figured that would save me from the GIL problems since each subprocess would have its own GIL.

In the words of Tom Waits, " the world just keeps getting bigger when you get out on your own". So I'm re-reading now, and maybe what I've been doing would have been better served by the multiprocessing package.

I'm running python2.6 on FreeBSD with a dual quadcore cpu. Now my questions are: (1) what the heck should I be doing to get concurrent builds of the chapters, wait for them all to finish, and pick up processing the main job again? The separate chapter builds have no need for communication--they're autonomous.
(2) using threads with the target fn calling subprocess, a bad idea?
(3) should I study up on multiprocessing package and/or pprocessing?

thanks for your inputs,

You could adapt the threading solution I gave to multiprocessing; just
use the multiprocessing queue class instead of the threading queue
class, etc.
--
http://mail.python.org/mailman/listinfo/python-list

Reply via email to