Last time I tried to use sup[1], I posted about sup-sync crashing with various symptoms of memory exhaustion[2]. I've tried again (using git mainline), with similar results, but now I have a bit more to say about it.
I am running on a fairly low-memory virtual machine. I think some of the variability in what I was seeing before had to do with what other things were running. Sorry about not being aware of this before. In the following, I have pretty well controlled for other system memory use. All of these tests are done on the same mbox with all indices cleared. Some of the failures I see are out of memory when running gpg ("Errno::ENOMEM Exception: Cannot allocate memory - /usr/bin/gpg" ...). I stopped at that point in the debugger and found that at this point, the ruby backtick operator fails the same way on any command. Using strace, I saw a failure in clone: 20528 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0xb7cb3908) = -1 ENOMEM (Cannot allocate memory) top reports 2.5M of physical and 30M of swap free, so I don't really know why clone fails, but I guess there's not much you can do about that. Other failures were the result of sup blowing up on messages with large attachments. sup's memory use is many times the attachment size. Testing on an mbox with a single message with 4 ~5M (encoded size) attachments (total file size ~21M), sup goes up to ~150M. Accounting for baseline, that's about 6 times the file size. I hope that can be improved. FWIW, mutt never seems to try to load a whole attachment into memory. Using the ferret index, the memory (virtual as reported by Linux) behaviour of sup-sync is pretty good. It starts out 25M and levels out around 31M. It spikes from time to time, presumably because of large messages, but it comes right back. After processing the last message, it uses another 10M to finish up. Using the xapian index, things are different. It starts at 32M and steadily climbs to 77M after ~3500 messages, or around 1M every 100 messages. It does seem to climb faster at first and then more slowly. Either xapian is keeping a cache (but some searches suggest it doesn't), it's leaking memory, or it's allocating memory in a way that the the allocator can't reclaim the VM space. Any ideas? BTW, is there really no way to ask for ruby's heap size with (unpatched) ruby 1.8? Andrew [1] This is about the fourth time. I seem to be easily discouraged. [2] http://rubyforge.org/pipermail/sup-talk/2009-May/002171.html _______________________________________________ sup-talk mailing list sup-talk@rubyforge.org http://rubyforge.org/mailman/listinfo/sup-talk