Joman to add a little bit more to one of my previous mails about the
readFields methods
Have you ever had something like this?
public class MyWritable implements Writable {
private DoubleWritable doubleWritable;
private TupleWritable tupleWritable;
public void readFields(DataInput in) throws IOException {
doubleWritable.readFields(in);
tupleWritable.readFields(in);
}
public void write(DataOutput out) throws IOException {
doubleWritable.write(out);
tupleWritable.write(out);
}
}
On Sun, Oct 19, 2008 at 1:59 AM, Joman Chu <[EMAIL PROTECTED]> wrote:
> I've never used TupleWritable, so hopefully somebody else can help you
> with that.
> Joman Chu
> http://www.notatypewriter.com/
> AIM: ARcanUSNUMquam
>
>
>
> On Sun, Oct 19, 2008 at 4:40 AM, Yih Sun Khoo <[EMAIL PROTECTED]> wrote:
> > Also, I've noticed TupleWritable to be quite useful.
> > What are good techniques for using TupleWritable in a mapping phase for a
> > "list of Text" when you do not know the size of that "list" ahead of time
> >
> > Say I had a custom writable which implemented TupleWritable and the
> custom
> > writable contained a setter method
> > mycustomwritable.setTupleWritable( ... )
> >
> > Where the ellipsis is, there lies the TupleWritable. However I'm
> wondering
> > since TupleWritable can be constructed using TupleWritable(Writable[]),
> how
> > do I dynamically resize the Writable[] and add Text elements to it when I
> > don't know the size of the Writable[] very well. Does this make sense?
> >
> >
> > On Sun, Oct 19, 2008 at 1:32 AM, Yih Sun Khoo <[EMAIL PROTECTED]> wrote:
> >
> >> Let's say in the reduce phase your value happens to hold an
> >> ArrayListWritable
> >> In this example, value is of type ArrayListWritable
> >> Maybe I've not thought about this or done this before, but how does one
> >> "read data in from the DataInput stream" in the reduce phase so that the
> >> ArrayListWritable which is a value already passed to the reducer can be
> used
> >> as ArrayListWritable
> >>
> >>
> >> On Sun, Oct 19, 2008 at 1:25 AM, Joman Chu <[EMAIL PROTECTED]> wrote:
> >>
> >>> Since the ArrayListWritable extends ArrayList, you have access to all
> >>> the ArrayList methods as well. Once you read data in from the
> >>> DataInput stream, you should be able to use ArrayListWritable just
> >>> like a regular ArrayList.
> >>> Joman Chu
> >>> http://www.notatypewriter.com/
> >>> AIM: ARcanUSNUMquam
> >>>
> >>>
> >>>
> >>> On Sun, Oct 19, 2008 at 4:16 AM, Yih Sun Khoo <[EMAIL PROTECTED]>
> wrote:
> >>> > Hmm, what method from ArrayListWritable allows you to access the
> >>> different
> >>> > elements of the ArrayList? Would it be readFields? for example, in
> a
> >>> > reduce phase, if I needed to know the size of the array list, it
> would
> >>> be
> >>> > easy if i were dealing with an arraylist because i could just say
> >>> > arraylist.size. How would i accomplish that with the writable
> >>> counterpart?
> >>> >
> >>> > On Sun, Oct 19, 2008 at 1:04 AM, Joman Chu <[EMAIL PROTECTED]>
> wrote:
> >>> >
> >>> >> Hi,
> >>> >>
> >>> >> For the ArrayList object, try taking a look at the implementation of
> >>> >> ArrayListWritable by Jimmy Lin at UMD here:
> >>> >>
> >>> >>
> >>> >>
> >>>
> https://subversion.umiacs.umd.edu/umd-hadoop/core/trunk/src/edu/umd/cloud9/io/ArrayListWritable.java
> >>> >>
> >>> >> But basically in the readFields methods, I prefer using each
> Writable
> >>> >> object's readFields method to read the data in. For example, for
> your
> >>> >> double variable, I would use a DoubleWritable object and in the
> >>> >> MyWritable.readFields(DataInput in), I would use
> >>> >> nameofdoublewritable.readFields(in). For the
> >>> >> MyWritable.write(DataOutput out) method, I would use
> >>> >> nameofdoublewritable.write(out).
> >>> >>
> >>> >> Have a good one,
> >>> >>
> >>> >> Joman Chu
> >>> >> http://www.notatypewriter.com/
> >>> >> AIM: ARcanUSNUMquam
> >>> >>
> >>> >>
> >>> >>
> >>> >> On Sun, Oct 19, 2008 at 3:30 AM, Yih Sun Khoo <[EMAIL PROTECTED]>
> >>> wrote:
> >>> >> > I don't quite know how to write the read and write functions, but
> I
> >>> want
> >>> >> to
> >>> >> > write my own writable, which should have a DoubleWritable/double
> >>> value
> >>> >> > followed by a list of Strings/Text. This Writable will be used as
> a
> >>> >> value.
> >>> >> > Is the code below the best way to go about writing such a
> writable?
> >>> >> >
> >>> >> > import java.io.DataInput;
> >>> >> > import java.io.DataOutput;
> >>> >> > import java.io.EOFException;
> >>> >> > import java.io.IOException;
> >>> >> > import java.util.ArrayList;
> >>> >> >
> >>> >> > import org.apache.hadoop.io.Writable;
> >>> >> >
> >>> >> > public class MyWritable implements Writable {
> >>> >> > private double score;
> >>> >> > private ArrayList<String> nameList;
> >>> >> >
> >>> >> > public void setScore(double score) {
> >>> >> > this.score= score;
> >>> >> > }
> >>> >> >
> >>> >> > public void setNameList(ArrayList<String> nameList) {
> >>> >> > this.nameList= nameList;
> >>> >> > }
> >>> >> >
> >>> >> > public double getScore() {
> >>> >> > return score;
> >>> >> > }
> >>> >> >
> >>> >> > public ArrayList<String> getNameList() {
> >>> >> > return nameList;
> >>> >> > }
> >>> >> >
> >>> >> > public void readFields(DataInput in) throws IOException {
> >>> >> > score= in.readDouble();
> >>> >> > try {
> >>> >> > do {
> >>> >> > nameList.add(in.readUTF());
> >>> >> > } while (true);
> >>> >> > } catch (EOFException eofe) {
> >>> >> > // continue; done
> >>> >> > }
> >>> >> > }
> >>> >> >
> >>> >> > public void write(DataOutput out) throws IOException {
> >>> >> > out.writeDouble(score);
> >>> >> > for (String name: nameList) {
> >>> >> > out.writeUTF(name);
> >>> >> > }
> >>> >> > }
> >>> >> > }
> >>> >> >
> >>> >>
> >>> >
> >>>
> >>
> >>
> >
>
