On approximately 10/30/2008 6:26 AM, came the following characters from
the keyboard of Jesse Noller:
On Wed, Oct 29, 2008 at 8:05 PM, Glenn Linderman <[EMAIL PROTECTED]> wrote:
On approximately 10/29/2008 3:45 PM, came the following characters from the
keyboard of Patrick Stinson:
If you are dealing with "lots" of data like in video or sound editing,
you would just keep the data in shared memory and send the reference
over IPC to the worker process. Otherwise, if you marshal and send you
are looking at a temporary doubling of the memory footprint of your
app because the data will be copied, and marshaling overhead.
Right. Sounds, and is, easy, if the data is all directly allocated by the
application. But when pieces are allocated by 3rd party libraries, that use
the C-runtime allocator directly, then it becomes more difficult to keep
everything in shared memory.
One _could_ replace the C-runtime allocator, I suppose, but that could have
some adverse effects on other code, that doesn't need its data to be in
shared memory. So it is somewhat between a rock and a hard place.
By avoiding shared memory, such problems are sidestepped... until you run
smack into the GIL.
If you do not have shared memory: You don't need threads, ergo: You
don't get penalized by the GIL. Threads are only useful when you need
to have that requirement of large in-memory data structures shared and
modified by a pool of workers.
The whole point of this thread is to talk about large in-memory data
structures that are shared and modified by a pool of workers.
My reference to shared memory was specifically referring to the concept
of sharing memory between processes... a particular OS feature that is
called shared memory.
The need for sharing memory among a pool of workers is still the
premise. Threads do that automatically, without the need for the OS
shared memory feature, that brings with it the need for a special
allocator to allocate memory in the shared memory area vs the rest of
the address space.
Not to pick on you, particularly, Jesse, but this particular response
made me finally understand why there has been so much repetition of the
same issues and positions over and over and over in this thread: instead
of comprehending the whole issue, people are responding to small
fragments of it, with opinions that may be perfectly reasonable for that
fragment, but missing the big picture, or the explanation made when the
same issue was raised in a different sub-thread.
--
Glenn -- http://nevcal.com/
===========================
A protocol is complete when there is nothing left to remove.
-- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking
--
http://mail.python.org/mailman/listinfo/python-list
