The Hadoop internals should handle that for you. Once the execution enters the reduce() method in your class that implements the Reducer interface, all of the data should have been read into the appropriate objects by the reducer. Hadoop itself will use the readFields() method you wrote in your own Writable to accomplish this.
If you've ever used any of Hadoop's own Writables, such as DoubleWritable or IntWritable in a MapReduce job, it is exactly the same thing. Hadoop's internals will transparently read the data into the DoubleWritable or IntWritable object and present it to you in reduce(). Joman Chu http://www.notatypewriter.com/ AIM: ARcanUSNUMquam On Sun, Oct 19, 2008 at 4:32 AM, Yih Sun Khoo <[EMAIL PROTECTED]> wrote: > Let's say in the reduce phase your value happens to hold an > ArrayListWritable > In this example, value is of type ArrayListWritable > Maybe I've not thought about this or done this before, but how does one > "read data in from the DataInput stream" in the reduce phase so that the > ArrayListWritable which is a value already passed to the reducer can be used > as ArrayListWritable > > On Sun, Oct 19, 2008 at 1:25 AM, Joman Chu <[EMAIL PROTECTED]> wrote: > >> Since the ArrayListWritable extends ArrayList, you have access to all >> the ArrayList methods as well. Once you read data in from the >> DataInput stream, you should be able to use ArrayListWritable just >> like a regular ArrayList. >> Joman Chu >> http://www.notatypewriter.com/ >> AIM: ARcanUSNUMquam >> >> >> >> On Sun, Oct 19, 2008 at 4:16 AM, Yih Sun Khoo <[EMAIL PROTECTED]> wrote: >> > Hmm, what method from ArrayListWritable allows you to access the >> different >> > elements of the ArrayList? Would it be readFields? for example, in a >> > reduce phase, if I needed to know the size of the array list, it would be >> > easy if i were dealing with an arraylist because i could just say >> > arraylist.size. How would i accomplish that with the writable >> counterpart? >> > >> > On Sun, Oct 19, 2008 at 1:04 AM, Joman Chu <[EMAIL PROTECTED]> wrote: >> > >> >> Hi, >> >> >> >> For the ArrayList object, try taking a look at the implementation of >> >> ArrayListWritable by Jimmy Lin at UMD here: >> >> >> >> >> >> >> https://subversion.umiacs.umd.edu/umd-hadoop/core/trunk/src/edu/umd/cloud9/io/ArrayListWritable.java >> >> >> >> But basically in the readFields methods, I prefer using each Writable >> >> object's readFields method to read the data in. For example, for your >> >> double variable, I would use a DoubleWritable object and in the >> >> MyWritable.readFields(DataInput in), I would use >> >> nameofdoublewritable.readFields(in). For the >> >> MyWritable.write(DataOutput out) method, I would use >> >> nameofdoublewritable.write(out). >> >> >> >> Have a good one, >> >> >> >> Joman Chu >> >> http://www.notatypewriter.com/ >> >> AIM: ARcanUSNUMquam >> >> >> >> >> >> >> >> On Sun, Oct 19, 2008 at 3:30 AM, Yih Sun Khoo <[EMAIL PROTECTED]> wrote: >> >> > I don't quite know how to write the read and write functions, but I >> want >> >> to >> >> > write my own writable, which should have a DoubleWritable/double value >> >> > followed by a list of Strings/Text. This Writable will be used as a >> >> value. >> >> > Is the code below the best way to go about writing such a writable? >> >> > >> >> > import java.io.DataInput; >> >> > import java.io.DataOutput; >> >> > import java.io.EOFException; >> >> > import java.io.IOException; >> >> > import java.util.ArrayList; >> >> > >> >> > import org.apache.hadoop.io.Writable; >> >> > >> >> > public class MyWritable implements Writable { >> >> > private double score; >> >> > private ArrayList<String> nameList; >> >> > >> >> > public void setScore(double score) { >> >> > this.score= score; >> >> > } >> >> > >> >> > public void setNameList(ArrayList<String> nameList) { >> >> > this.nameList= nameList; >> >> > } >> >> > >> >> > public double getScore() { >> >> > return score; >> >> > } >> >> > >> >> > public ArrayList<String> getNameList() { >> >> > return nameList; >> >> > } >> >> > >> >> > public void readFields(DataInput in) throws IOException { >> >> > score= in.readDouble(); >> >> > try { >> >> > do { >> >> > nameList.add(in.readUTF()); >> >> > } while (true); >> >> > } catch (EOFException eofe) { >> >> > // continue; done >> >> > } >> >> > } >> >> > >> >> > public void write(DataOutput out) throws IOException { >> >> > out.writeDouble(score); >> >> > for (String name: nameList) { >> >> > out.writeUTF(name); >> >> > } >> >> > } >> >> > } >> >> > >> >> >> > >> >
