Malcolm Greene wrote: > I'm looking for tips on how to load balance running multiple Python > applications in multi-CPU environments. My understanding is that Python > applications and their threads are limited to a specific CPU. > > Background: I have a Python utility that processes email messages. I > suspect there's a lot of idle time while this utility waits on a remote > email server. I would like to run as many simultaneous copies of this > utility as possible without overwhelming the server these utilities are > running on. My thought is that I should write a dispatcher that monitors > CPU load and launches/cancels multiple instances of my utility with > specific job queues to process. > > Is there a cross-platform way to monitor CPU load? > > Is there a pre-existing Python module I should be looking at for > building (subclassing) my dispatcher? > As a data point for you, I wrote a multi-threaded application that used 100 threads to send about 45,000 emails in less than four hours. The single-CPU computer that processed this job wasn't using that much CPU (certainly under 50%).
regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 Holden Web LLC http://www.holdenweb.com/ -- http://mail.python.org/mailman/listinfo/python-list