Hi,
I am trying to use mulitpleOutputFormat to generate outputs to multiple
files. But the reducer fails with nullpointer
at MultipleOutputFormat.write().
Following is the Job conf:
public class OutPutFileNameGenerator extends MultipleTextOutputFormat<Text,
Text>{
protected String generateFileNameForKeyValue(Text key, Text value,
String name) {
System.out.println( "Key is : " + key );
String str = "partitionFor_" + key.toString();
return str ;
}
}
@Override
public int run(String[] args) throws Exception {
Configuration conf = getConf();
JobConf job = new JobConf(conf, DataPartitioner.class);
Path in = new Path(args[0]);
Path out = new Path(args[1]);
FileInputFormat.setInputPaths(job, in);
FileOutputFormat.setOutputPath(job, out);
job.setJobName("Data Partitioner");
job.setMapperClass(PartitionMapper.class);
job.setReducerClass(PartitionReducer.class);
job.setInputFormat(TextInputFormat.class);
job.setOutputFormat(OutPutFileNameGenerator.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(Text.class);
// job.setNumReduceTasks(0);
JobClient.runJob(job);
return 0;
}
Stack Trace:
3/08/10 17:25:11 INFO mapred.JobClient: map 100% reduce 0%
13/08/10 17:25:21 INFO mapred.JobClient: map 100% reduce 33%
13/08/10 17:25:24 INFO mapred.JobClient: Task Id :
attempt_201308101504_0009_r_000000_1, Status : FAILED
java.lang.NullPointerException
at
org.apache.hadoop.mapred.lib.MultipleOutputFormat$1.write(MultipleOutputFormat.java:102)
at
org.apache.hadoop.mapred.ReduceTask$OldTrackingRecordWriter.write(ReduceTask.java:458)
at
org.apache.hadoop.mapred.ReduceTask$3.collect(ReduceTask.java:498)
at org.partitioner.PartitionReducer.reduce(PartitionReducer.java:24)
at org.partitioner.PartitionReducer.reduce(PartitionReducer.java:1)
at
org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:520)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:421)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
--
- Ravi
--
- Ravi