Based on some similar problems that I found others were having in the mailing lists, it looks like the solution was to list my Map/Reduce job JAR In the conf/hadoop-env.sh file under HADOOP_CLASSPATH. After doing that and re-submitting the job, it all worked fine! I guess the MapWritable class somehow doesn't share the same classpath as the program that actually submits the job conf. Is this expected?
Thanks, Ryan On Tue, Sep 9, 2008 at 9:44 AM, Ryan LeCompte <[EMAIL PROTECTED]> wrote: > Okay, I think I'm getting closer but now I'm running into another problem. > > First off, I created my own CustomMapWritable that extends MapWritable > and invokes AbstractMapWritable.addToMap() to add my custom classes. > Now the map/reduce phases actually complete and the job as a whole > completes. However, when I try to use the SequenceFile API to later > read the output data, I'm getting a strange exception. First the code: > > FileSystem fileSys = FileSystem.get(conf); > SequenceFile.Reader reader = new SequenceFile.Reader(fileSys, inFile, > conf); > Text key = new Text(); > CustomWritable stats = new CustomWritable(); > reader.next(key, stats); > reader.close(); > > And now the exception that's thrown: > > java.io.IOException: can't find class: com.test.CustomStatsWritable > because com.test.CustomStatsWritable > at > org.apache.hadoop.io.AbstractMapWritable.readFields(AbstractMapWritable.java:210) > at org.apache.hadoop.io.MapWritable.readFields(MapWritable.java:145) > at com.test.CustomStatsWritable.readFields(UserStatsWritable.java:49) > at > org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.java:1751) > at > org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1879) > ... > > Any ideas? > > Thanks, > Ryan > > > On Tue, Sep 9, 2008 at 12:36 AM, Ryan LeCompte <[EMAIL PROTECTED]> wrote: >> Hello, >> >> I'm attempting to use a SortedMapWritable with a LongWritable as the >> key and a custom implementation of org.apache.hadoop.io.Writable as >> the value. I notice that my program works fine when I use another >> primitive wrapper (e.g. Text) as the value, but fails with the >> following exception when I use my custom Writable instance: >> >> 2008-09-08 23:25:02,072 INFO org.apache.hadoop.mapred.ReduceTask: >> Initiating in-memory merge with 1 segments... >> 2008-09-08 23:25:02,077 INFO org.apache.hadoop.mapred.Merger: Merging >> 1 sorted segments >> 2008-09-08 23:25:02,077 INFO org.apache.hadoop.mapred.Merger: Down to >> the last merge-pass, with 1 segments left of total size: 5492 bytes >> 2008-09-08 23:25:02,099 WARN org.apache.hadoop.mapred.ReduceTask: >> attempt_200809082247_0005_r_000000_0 Merge of the inmemory files threw >> a >> n exception: java.io.IOException: Intermedate merge failed >> at >> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.doInMemMerge(ReduceTask.java:2133) >> at >> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.run(ReduceTask.java:2064) >> Caused by: java.lang.RuntimeException: java.lang.NullPointerException >> at >> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:80) >> at >> org.apache.hadoop.io.SortedMapWritable.readFields(SortedMapWritable.java:179) >> ... >> >> I noticed that the AbstractMapWritable class has a protected >> "addToMap(Class clazz)" method. Do I somehow need to let my >> SortedMapWritable instance know about my custom Writable value? I've >> properly implemented the custom Writable object (it just contains a >> few primitives, like longs and ints). >> >> Any insight is appreciated. >> >> Thanks, >> Ryan >> >
