Thanks for your information, Todd. It's cool to address the job overhead in JVM reuse way. You mentioned that the code(and variables?) are kept in JIT cache, so can we refer the variables in the last iteration in some way? I am really caurious how this works in coding level. Could you please help to point out some material/resources I can start to get to know the usage of this feature? Is this implemented in the current 0.20 version? I am pretty fresh to haoop. :-)
Thanks, Jianmin ________________________________ From: Todd Lipcon <[email protected]> To: [email protected] Sent: Tuesday, June 2, 2009 1:36:17 PM Subject: Re: question about when shuffle/sort start working On Mon, Jun 1, 2009 at 7:49 PM, Jianmin Woo <[email protected]> wrote: > > In this way, if some job fails, we can re-run the whole job sequence > starting from the latest checkpoint instead of the beginning. It will be > nice if all the sequence (it`s actually a loop which happens in many of the > machine learning algorithms, each loop contains a Map and Reduce step. > PageRank calculation is one of the examples) can be done in a single job. > Because if an algorithm takes 1000 steps to converge, we have to start 1000 > jobs in the job sequence way, which is costly since of the start/stop of > jobs. Yep - as you mentioned this is very common in many algorithms on MapReduce, especially in machine learning and graph processing. The only way to do this is, like you said, run 1000 jobs. Job overhead is one thing that has been improved significantly in recent versions. JVM reuse is one such new-ish feature that helps here, since the overhead of spawning tasktracker children is eliminated, and your code stays in the JIT cache between iterations. -Todd
