Also, I've noticed TupleWritable to be quite useful. What are good techniques for using TupleWritable in a mapping phase for a "list of Text" when you do not know the size of that "list" ahead of time
Say I had a custom writable which implemented TupleWritable and the custom writable contained a setter method mycustomwritable.setTupleWritable( ... ) Where the ellipsis is, there lies the TupleWritable. However I'm wondering since TupleWritable can be constructed using TupleWritable(Writable[]), how do I dynamically resize the Writable[] and add Text elements to it when I don't know the size of the Writable[] very well. Does this make sense? On Sun, Oct 19, 2008 at 1:32 AM, Yih Sun Khoo <[EMAIL PROTECTED]> wrote: > Let's say in the reduce phase your value happens to hold an > ArrayListWritable > In this example, value is of type ArrayListWritable > Maybe I've not thought about this or done this before, but how does one > "read data in from the DataInput stream" in the reduce phase so that the > ArrayListWritable which is a value already passed to the reducer can be used > as ArrayListWritable > > > On Sun, Oct 19, 2008 at 1:25 AM, Joman Chu <[EMAIL PROTECTED]> wrote: > >> Since the ArrayListWritable extends ArrayList, you have access to all >> the ArrayList methods as well. Once you read data in from the >> DataInput stream, you should be able to use ArrayListWritable just >> like a regular ArrayList. >> Joman Chu >> http://www.notatypewriter.com/ >> AIM: ARcanUSNUMquam >> >> >> >> On Sun, Oct 19, 2008 at 4:16 AM, Yih Sun Khoo <[EMAIL PROTECTED]> wrote: >> > Hmm, what method from ArrayListWritable allows you to access the >> different >> > elements of the ArrayList? Would it be readFields? for example, in a >> > reduce phase, if I needed to know the size of the array list, it would >> be >> > easy if i were dealing with an arraylist because i could just say >> > arraylist.size. How would i accomplish that with the writable >> counterpart? >> > >> > On Sun, Oct 19, 2008 at 1:04 AM, Joman Chu <[EMAIL PROTECTED]> wrote: >> > >> >> Hi, >> >> >> >> For the ArrayList object, try taking a look at the implementation of >> >> ArrayListWritable by Jimmy Lin at UMD here: >> >> >> >> >> >> >> https://subversion.umiacs.umd.edu/umd-hadoop/core/trunk/src/edu/umd/cloud9/io/ArrayListWritable.java >> >> >> >> But basically in the readFields methods, I prefer using each Writable >> >> object's readFields method to read the data in. For example, for your >> >> double variable, I would use a DoubleWritable object and in the >> >> MyWritable.readFields(DataInput in), I would use >> >> nameofdoublewritable.readFields(in). For the >> >> MyWritable.write(DataOutput out) method, I would use >> >> nameofdoublewritable.write(out). >> >> >> >> Have a good one, >> >> >> >> Joman Chu >> >> http://www.notatypewriter.com/ >> >> AIM: ARcanUSNUMquam >> >> >> >> >> >> >> >> On Sun, Oct 19, 2008 at 3:30 AM, Yih Sun Khoo <[EMAIL PROTECTED]> >> wrote: >> >> > I don't quite know how to write the read and write functions, but I >> want >> >> to >> >> > write my own writable, which should have a DoubleWritable/double >> value >> >> > followed by a list of Strings/Text. This Writable will be used as a >> >> value. >> >> > Is the code below the best way to go about writing such a writable? >> >> > >> >> > import java.io.DataInput; >> >> > import java.io.DataOutput; >> >> > import java.io.EOFException; >> >> > import java.io.IOException; >> >> > import java.util.ArrayList; >> >> > >> >> > import org.apache.hadoop.io.Writable; >> >> > >> >> > public class MyWritable implements Writable { >> >> > private double score; >> >> > private ArrayList<String> nameList; >> >> > >> >> > public void setScore(double score) { >> >> > this.score= score; >> >> > } >> >> > >> >> > public void setNameList(ArrayList<String> nameList) { >> >> > this.nameList= nameList; >> >> > } >> >> > >> >> > public double getScore() { >> >> > return score; >> >> > } >> >> > >> >> > public ArrayList<String> getNameList() { >> >> > return nameList; >> >> > } >> >> > >> >> > public void readFields(DataInput in) throws IOException { >> >> > score= in.readDouble(); >> >> > try { >> >> > do { >> >> > nameList.add(in.readUTF()); >> >> > } while (true); >> >> > } catch (EOFException eofe) { >> >> > // continue; done >> >> > } >> >> > } >> >> > >> >> > public void write(DataOutput out) throws IOException { >> >> > out.writeDouble(score); >> >> > for (String name: nameList) { >> >> > out.writeUTF(name); >> >> > } >> >> > } >> >> > } >> >> > >> >> >> > >> > >
