At 11:46 PM 3/12/02 +0800, Stas Bekman wrote: >>I'm not sure whether my assessment of the problem is correct. I would >>welcome any comments on this. >Nope Elizabeth, your explanation is not so correct. ;)
Too bad... ;-( >Shared memory is not about sharing the pre-allocated memory pool (heap >memory). Once you re-use a bit of preallocated memory the sharing goes away. I think the phrase is Copy-On-Write, right? And since RAM is allocated in chunks, let's assume 4K for the sake of the argument, changing a single byte in such a chunk causes the entire chunk to be unshared. In older Linux kernels, I believe to have seen that when a byte gets changed in a chunk of any child, that chunk becomes changed for _all_ children. Newer kernels only unshare it for that particular child. Again, if I'm not mistaken and someone please correct me if I'm wrong... Since Perl is basically all data, you would need to find a way of localizing all memory that is changing to as few memory chunks as possible. My idea was just that: by filling up all "used" memory before spawning children, you would use op some memory, but that would be shared between all children and thus not so bad. But by doing this, you would hopefully cause all changing data to be localized to newly allocated memory by the children. Wish someone with more Perl guts experience could tell me if that really is an idea that could work or not... >Shared memory is about 'text'/read-only memory pages which never get >modified and pages that can get modified but as long as they aren't >modified they are shared. Unfortunately (in this aspect, but fortunately >for many other aspects) Perl is not a strongly-typed (or whatever you call >it) language, therefore it's extremely hard to share memory, because in >Perl almost everything is data. Though as you could see Bill was able to >share 43M out of 50M which is damn good! As a proof of concept I have run more than 100 200MB+ children on a 1 GB RAM machine and had sharing go up so high causing the "top" number of bytes field for shared memory to cycle through its 32-bit range multiple times... ;-) . It was _real_ fast (had all of its data that it needed as Perl hashes and lists) and ran ok until something would start an avalanche effect and it would all go down in a whirlwind of swapping. So in the end, it didn't work reliably enough ;-( But man, was it fast when it ran... ;-) Elizabeth Mattijsen
