Thanks Ted. Is it correct to assume that all class members defined
inside my Mapper are visible to all of the threads, so I should pay
careful attention and take synchronization into account when accessing
those objects?
Jim
On Tue, Apr 27, 2010 at 11:50 PM, Ted Yu yuzhih...@gmail.com wrote:
Looking through MultithreadedMapRunner, map() seems to be the only method
called by executorService:
MultithreadedMapRunner.this.mapper.map(key, value, output,
reporter);
On Tue, Apr 27, 2010 at 3:46 PM, Jim Twensky jim.twen...@gmail.com wrote:
Hi,
I've decided to refactor some of my Hadoop jobs and implement them
using MultithreadedMapper.class but I got puzzled because of some
unexpected error messages at run time.
Here are some relevant settings regarding my Hadoop cluster:
mapred.tasktracker.map.tasks.maximum = 1
mapred.tasktracker.reduce.tasks.maximum = 1
mapred.job.reuse.jvm.num.tasks = -1
mapred.map.multithreadedrunner.threads = 4
I'd like to know how threads are used to run the map task in a single
JVM (Correct me if this is wrong). Suppose I've got a sample Mapper
class as such:
class Mapper ... {
MyObject A;
static MyObject B;
setup() {
Configuration conf = context.getConfiguration();
A.initialize(c);
B.initialize(c);
}
map() {...}
cleanup() {...}
Does each thread run all three of setup(), map(), cleanup() methods ?
-OR-
Are setup() and cleanup() run once per task (and thus per JVM
according to my settings) and so map is the only multithreaded
function?
Also, are the objects A and B shared among different threads or does
each trade have its own copy of them? My initial guess was that each
thread would have a separate copy of A, and B would be shared among
the 4 threads running on the same box since it is defined as static,
but it appears to me that this assumption is not correct and A seems
to be shared.
Thanks,
Jim
