Speaking of the big picture, is this how it normally works when someone says "Here's some code and a problem and I'm willing to pay for a solution?" I've never really walked that path with a project of this complexity (I guess it's the backwards-compatibility that makes it confusing), but is this problem just too complex so we have to keep talking and talking on forum after forum? Afraid to fork? I know I am. How many people are qualified to tackle Andy's problem? Are all of them busy or uninterested? Is the current code in a tight spot where it just can't be fixed without really jabbing that FORK in so deep that the patch will die when your project does?
Personally I think this problem is super-awesome on the hobbyest's fun scale. I'd totally take the time to let my patch do the talking but I haven't read enough of the (2.5) code. So, I resort to simply reading the newsgroups and python code to better understand the mechanics problem :( On Thu, Oct 30, 2008 at 2:54 PM, Glenn Linderman <[EMAIL PROTECTED]> wrote: > On approximately 10/30/2008 6:26 AM, came the following characters from the > keyboard of Jesse Noller: >> >> On Wed, Oct 29, 2008 at 8:05 PM, Glenn Linderman <[EMAIL PROTECTED]> >> wrote: >> >>> >>> On approximately 10/29/2008 3:45 PM, came the following characters from >>> the >>> keyboard of Patrick Stinson: >>> >>>> >>>> If you are dealing with "lots" of data like in video or sound editing, >>>> you would just keep the data in shared memory and send the reference >>>> over IPC to the worker process. Otherwise, if you marshal and send you >>>> are looking at a temporary doubling of the memory footprint of your >>>> app because the data will be copied, and marshaling overhead. >>>> >>> >>> Right. Sounds, and is, easy, if the data is all directly allocated by >>> the >>> application. But when pieces are allocated by 3rd party libraries, that >>> use >>> the C-runtime allocator directly, then it becomes more difficult to keep >>> everything in shared memory. >>> >>> One _could_ replace the C-runtime allocator, I suppose, but that could >>> have >>> some adverse effects on other code, that doesn't need its data to be in >>> shared memory. So it is somewhat between a rock and a hard place. >>> >>> By avoiding shared memory, such problems are sidestepped... until you run >>> smack into the GIL. >>> >> >> If you do not have shared memory: You don't need threads, ergo: You >> don't get penalized by the GIL. Threads are only useful when you need >> to have that requirement of large in-memory data structures shared and >> modified by a pool of workers. > > The whole point of this thread is to talk about large in-memory data > structures that are shared and modified by a pool of workers. > > My reference to shared memory was specifically referring to the concept of > sharing memory between processes... a particular OS feature that is called > shared memory. > > The need for sharing memory among a pool of workers is still the premise. > Threads do that automatically, without the need for the OS shared memory > feature, that brings with it the need for a special allocator to allocate > memory in the shared memory area vs the rest of the address space. > > Not to pick on you, particularly, Jesse, but this particular response made > me finally understand why there has been so much repetition of the same > issues and positions over and over and over in this thread: instead of > comprehending the whole issue, people are responding to small fragments of > it, with opinions that may be perfectly reasonable for that fragment, but > missing the big picture, or the explanation made when the same issue was > raised in a different sub-thread. > > -- > Glenn -- http://nevcal.com/ > =========================== > A protocol is complete when there is nothing left to remove. > -- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking > > -- http://mail.python.org/mailman/listinfo/python-list