LocalJobRunner allows you to test your code with everything running in a
single JVM. Just set mapred.job.tracker=local.
Doug
Ryan LeCompte wrote:
I see... so there really isn't a way for me to test a map/reduce
program using a single node without incurring the overhead of
upping/downing JVM's... My input is broken up into 5 text files.... is
there a way I could start the job such that it only uses 1 map to
process the whole thing? I guess I'd have to concatenate the files
into 1 file and somehow turn off splitting?
Ryan
On Wed, Sep 3, 2008 at 12:09 AM, Owen O'Malley <[EMAIL PROTECTED]> wrote:
On Sep 2, 2008, at 9:00 PM, Ryan LeCompte wrote:
Beginner's question:
If I have a cluster with a single node that has a max of 1 map/1
reduce, and the job submitted has 50 maps... Then it will process only
1 map at a time. Does that mean that it's spawning 1 new JVM for each
map processed? Or re-using the same JVM when a new map can be
processed?
It creates a new JVM for each task. Devaraj is working on
https://issues.apache.org/jira/browse/HADOOP-249
which will allow the jvms to run multiple tasks sequentially.
-- Owen
