Distributed Cache

Hi, I'm trying to use Distributed cache to add a jar file to the job and I'm using the following code, but it didn't work. If you have used it I would appreciate if you can post a sample Thank you Raja Thiruvathuru Configuration conf = new Configuration(); Job job = new Job(conf, "MyDriv

Re: log files not found

Hi all, I have found the log files on the DataNodes. I have checked the userlogs, but they do not contain any exception related to the error I have mentioned in the previous email (I am putting it here again). 10/04/01 01:04:15 INFO mapred.JobClient: Task Id : attempt_201003240138_0110_r_ 0

Partitioning Reducer Output

Hi, What's the best way to partition data generated from Reducer into multiple = directories in Hadoop 0.20.1. I was thinking of using MultipleTextOutputFor= mat but that's not backward compatible with other API's in this version of = hadoop. Thanks, -Rakesh

Re: Reflective instantiation of Mappers and Reducers

Or heck... I could just base-64 encode the serialized byte arrays and pass them as strings in the configuration. If it's going to be a hack, might as well go all the way. On Fri, Apr 2, 2010 at 4:10 PM, Kris Nuttycombe wrote: > On Fri, Apr 2, 2010 at 3:10 PM, Owen O'Malley wrote: >> >> On Apr 2,

Re: Reflective instantiation of Mappers and Reducers

On Fri, Apr 2, 2010 at 3:10 PM, Owen O'Malley wrote: > > On Apr 2, 2010, at 12:05 PM, Kris Nuttycombe wrote: > >> What I'm wondering is, is there any way to simply serialize a Mapper >> or Reducer object, and have the serialized instance copied, passed >> around and used everywhere instead of alwa

Re: Reflective instantiation of Mappers and Reducers

On Apr 2, 2010, at 12:05 PM, Kris Nuttycombe wrote: What I'm wondering is, is there any way to simply serialize a Mapper or Reducer object, and have the serialized instance copied, passed around and used everywhere instead of always having the Mapper and Reducer instantiated by reflection? This

Reflective instantiation of Mappers and Reducers

Hi, all, I'm new to Hadoop, and I'm finding myself having a hard time creating highly configurable Mapper and Reducer instances, due to the fact that Hadoop seems to require that Mapper and Reducer instances be instantiated through reflection, so I'm sort of wondering whether I'm doing things wro

What's the purpose of a setup and cleanup task?

Hi, I've posted this post last moth, but I haven't got a response. Does anyone knows this question? I would like to understand what's the purpose of a setup and cleanup task. During the start-up of the job tracker, it will be assigned 2 setup tasks and 2 cleanup tasks for map and for the reduce.