perhaps the java process can have a signal handler to checkpoint in memory data and then suspend itself? what i mean is it could finish any intermediate sort/merge runs and then suspend. next time it restarts, it would, hopefully, just wouldn't need much of the memory from it's data segment and could start afresh. but this is speculating that most of the memory consumption is sort buffering ..
________________________________ From: Doug Cutting [mailto:[EMAIL PROTECTED] Sent: Thu 1/10/2008 2:39 PM To: hadoop-user@lucene.apache.org Subject: Re: Question on running simultaneous jobs Joydeep Sen Sarma wrote: > being paged out is sad - but the worst case is still no worse than killing > the job (where all the data has to be *recomputed* back into memory on > restart - not just swapped in from disk) In my experience, once a large process is paged out it is almost always faster to restart it than to wait for it to get paged back in with random disk accesses. If there were a way to explicitly write out a process's working set, and then restore it later, using sequential disk accesses, that might be effective. Virtualization systems support such operations, so perhaps tasktrackers should start a Xen instance per task? Doug
