On Dec 29, 6:05 pm, "James Mills" <prolo...@shortcircuit.net.au> wrote: > On Tue, Dec 30, 2008 at 12:52 AM, mk <mrk...@gmail.com> wrote: > > Hello everyone, > > > After readinghttp://www.python.org/dev/peps/pep-0371/I was under > > impression that performance of multiprocessing package is similar to that of > > thread / threading. However, to familiarize myself with both packages I > > wrote my own test of spawning and returning 100,000 empty threads or > > processes (while maintaining at most 100 processes / threads active at any > > one time), respectively. snip > As others have mentioned, it's not suprising > that spawning even 100 processes took some > time. > > Bottom line: multiprocessing should not be used this way. > (nor should threading).
The OP may be interested in Erlang, which Wikipedia (end-all, be-all) claims is a 'distribution oriented language'. You might also find it interesting to examine a theoretical OS that is optimized for process overhead. In other words, what is the minimum overhead possible? Can processes be as small as threads? Can entire threads be only a few bytes (words) big? Also, could generators provide any of the things you need with your multiple threads? You could, say, call 'next()' on many items in a list, and just remove them on StopIteration. -- http://mail.python.org/mailman/listinfo/python-list
