error while using ArrayWritable and how to fix it?
Hello everyone,
I want to find out how to get an array as output in mapreduce.
I just simply modified the reduce method of wordcount example as shown below:
public static class IntSumReducer extends
Reducer {
private IntWritable[] iw = new IntWritable[2];
private ArrayWritable result = new ArrayWritable
(IntWritable.class,iw);
public void reduce(Text key, Iterable values,
Context context) throws IOException,
InterruptedException {
iw[0] = new IntWritable(); // initialize
iw[1] = new IntWritable(); // initialize
int sum = 0;
for (IntWritable val : values) {
sum += val.get();
}
iw[0].set(sum);
iw[1].set(sum);
result.set(iw);
context.write(key, result);
}
}
Map outputs .
Reduce takes > as inputs.
The moment I'm trying to use the ArrayWritable in reduce to store the results
I get the following error :
11/03/08 17:03:07 INFO jvm.JvmMetrics: Initializing JVM Metrics with
processName=JobTracker, sessionId=
11/03/08 17:03:07 WARN mapred.JobClient: No job jar file set. User classes
may not be found. See JobConf(Class) or JobConf#setJar(String).
11/03/08 17:03:07 INFO input.FileInputFormat: Total input paths to process : 1
11/03/08 17:03:07 INFO mapred.JobClient: Running job: job_local_0001
11/03/08 17:03:07 INFO input.FileInputFormat: Total input paths to process : 1
11/03/08 17:03:07 INFO mapred.MapTask: io.sort.mb = 100
11/03/08 17:03:07 INFO mapred.MapTask: data buffer = 79691776/99614720
11/03/08 17:03:07 INFO mapred.MapTask: record buffer = 262144/327680
11/03/08 17:03:08 INFO mapred.MapTask: Starting flush of map output
11/03/08 17:03:08 WARN mapred.LocalJobRunner: job_local_0001
java.io.IOException: wrong value class: class
org.apache.hadoop.io.ArrayWritable is not class
org.apache.hadoop.io.IntWritable
at org.apache.hadoop.mapred.IFile$Writer.append(IFile.java:167)
at org.apache.hadoop.mapred.Task$CombineOutputCollector.collect
(Task.java:880)
at
org.apache.hadoop.mapred.Task$NewCombinerRunner$OutputConverter.write
(Task.java:1201)
at org.apache.hadoop.mapreduce.TaskInputOutputContext.write
(TaskInputOutputContext.java:80)
at org.apache.hadoop.examples.WordCount$IntSumReducer.reduce
(WordCount.java:59)
at org.apache.hadoop.examples.WordCount$IntSumReducer.reduce
(WordCount.java:1)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:176)
at org.apache.hadoop.mapred.Task$NewCombinerRunner.combine
(Task.java:1222)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill
(MapTask.java:1265)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush
(MapTask.java:1129)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close
(MapTask.java:549)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:623)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run
(LocalJobRunner.java:177)
11/03/08 17:03:08 INFO mapred.JobClient: map 0% reduce 0%
11/03/08 17:03:08 INFO mapred.JobClient: Job complete: job_local_0001
11/03/08 17:03:08 INFO mapred.JobClient: Counters: 0
Re: error while using ArrayWritable
Thanks J-D , worked perfectly fine ! On Sat, Jan 2, 2010 at 1:16 PM, Jean-Daniel Cryans wrote: > This is explained in the javadoc: > > > http://hadoop.apache.org/common/docs/r0.20.1/api/org/apache/hadoop/io/ArrayWritable.html > > J-D > > On Fri, Jan 1, 2010 at 11:29 PM, bharath vissapragada > wrote: > > Hi all , > > > > I am using ArrayWritable in my MR job . > > > > Map outputs > > > > Reduce takes > > > > > The moment Iam trying to use the ArrayWritable in reduce using the > iterator > > ..I get the following error : > > > > 10/01/02 18:23:41 WARN mapred.LocalJobRunner: job_local_0001 > > java.lang.RuntimeException: java.lang.NoSuchMethodException: > > org.apache.hadoop.io.ArrayWritable.() > >at > > > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:81) > >at > > > org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:62) > >at > > > org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:40) > >at > > org.apache.hadoop.mapred.Task$ValuesIterator.readNextValue(Task.java:940) > >at org.apache.hadoop.mapred.Task$ValuesIterator.next(Task.java:880) > >at > > > org.apache.hadoop.mapred.ReduceTask$ReduceValuesIterator.moveToNext(ReduceTask.java:237) > >at > > > org.apache.hadoop.mapred.ReduceTask$ReduceValuesIterator.next(ReduceTask.java:233) > >at HashJoin.MR_hash$redu.reduce(MR_hash.java:132) > >at mtr.MyTableReduce.reduce(MyTableReduce.java:1) > >at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:436) > >at > > org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:170) > > Caused by: java.lang.NoSuchMethodException: > > org.apache.hadoop.io.ArrayWritable.() > >at java.lang.Class.getConstructor0(Class.java:2723) > >at java.lang.Class.getDeclaredConstructor(Class.java:2002) > >at > > > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:75) > >... 10 more > > java.io.IOException: Job failed! > >at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232) > >at HashJoin.MR_hash.run(MR_hash.java:294) > >at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) > >at HashJoin.MR_hash.join(MR_hash.java:320) > >at HashJoin.MR_hash.main(MR_hash.java:338) > > > > > > Can Anyone tell me what the error is ?? > > >
Re: error while using ArrayWritable
This is explained in the javadoc: http://hadoop.apache.org/common/docs/r0.20.1/api/org/apache/hadoop/io/ArrayWritable.html J-D On Fri, Jan 1, 2010 at 11:29 PM, bharath vissapragada wrote: > Hi all , > > I am using ArrayWritable in my MR job . > > Map outputs > > Reduce takes > > > The moment Iam trying to use the ArrayWritable in reduce using the iterator > ..I get the following error : > > 10/01/02 18:23:41 WARN mapred.LocalJobRunner: job_local_0001 > java.lang.RuntimeException: java.lang.NoSuchMethodException: > org.apache.hadoop.io.ArrayWritable.() > at > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:81) > at > org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:62) > at > org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:40) > at > org.apache.hadoop.mapred.Task$ValuesIterator.readNextValue(Task.java:940) > at org.apache.hadoop.mapred.Task$ValuesIterator.next(Task.java:880) > at > org.apache.hadoop.mapred.ReduceTask$ReduceValuesIterator.moveToNext(ReduceTask.java:237) > at > org.apache.hadoop.mapred.ReduceTask$ReduceValuesIterator.next(ReduceTask.java:233) > at HashJoin.MR_hash$redu.reduce(MR_hash.java:132) > at mtr.MyTableReduce.reduce(MyTableReduce.java:1) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:436) > at > org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:170) > Caused by: java.lang.NoSuchMethodException: > org.apache.hadoop.io.ArrayWritable.() > at java.lang.Class.getConstructor0(Class.java:2723) > at java.lang.Class.getDeclaredConstructor(Class.java:2002) > at > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:75) > ... 10 more > java.io.IOException: Job failed! > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232) > at HashJoin.MR_hash.run(MR_hash.java:294) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) > at HashJoin.MR_hash.join(MR_hash.java:320) > at HashJoin.MR_hash.main(MR_hash.java:338) > > > Can Anyone tell me what the error is ?? >
error while using ArrayWritable
Hi all , I am using ArrayWritable in my MR job . Map outputs Reduce takes > The moment Iam trying to use the ArrayWritable in reduce using the iterator ..I get the following error : 10/01/02 18:23:41 WARN mapred.LocalJobRunner: job_local_0001 java.lang.RuntimeException: java.lang.NoSuchMethodException: org.apache.hadoop.io.ArrayWritable.() at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:81) at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:62) at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:40) at org.apache.hadoop.mapred.Task$ValuesIterator.readNextValue(Task.java:940) at org.apache.hadoop.mapred.Task$ValuesIterator.next(Task.java:880) at org.apache.hadoop.mapred.ReduceTask$ReduceValuesIterator.moveToNext(ReduceTask.java:237) at org.apache.hadoop.mapred.ReduceTask$ReduceValuesIterator.next(ReduceTask.java:233) at HashJoin.MR_hash$redu.reduce(MR_hash.java:132) at mtr.MyTableReduce.reduce(MyTableReduce.java:1) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:436) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:170) Caused by: java.lang.NoSuchMethodException: org.apache.hadoop.io.ArrayWritable.() at java.lang.Class.getConstructor0(Class.java:2723) at java.lang.Class.getDeclaredConstructor(Class.java:2002) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:75) ... 10 more java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232) at HashJoin.MR_hash.run(MR_hash.java:294) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at HashJoin.MR_hash.join(MR_hash.java:320) at HashJoin.MR_hash.main(MR_hash.java:338) Can Anyone tell me what the error is ??
