On Mon, Jun 1, 2009 at 7:49 PM, Jianmin Woo <[email protected]> wrote:
> > In this way, if some job fails, we can re-run the whole job sequence > starting from the latest checkpoint instead of the beginning. It will be > nice if all the sequence (it`s actually a loop which happens in many of the > machine learning algorithms, each loop contains a Map and Reduce step. > PageRank calculation is one of the examples) can be done in a single job. > Because if an algorithm takes 1000 steps to converge, we have to start 1000 > jobs in the job sequence way, which is costly since of the start/stop of > jobs. Yep - as you mentioned this is very common in many algorithms on MapReduce, especially in machine learning and graph processing. The only way to do this is, like you said, run 1000 jobs. Job overhead is one thing that has been improved significantly in recent versions. JVM reuse is one such new-ish feature that helps here, since the overhead of spawning tasktracker children is eliminated, and your code stays in the JIT cache between iterations. -Todd
