Hey Joman,
public void readFields(DataInput in) throws IOException {
doubleWritable.readFields(in);
tupleWritable.readFields(in);
}
It seems this line of code is giving me some problems
doubleWritable.readFields(in);
The error stack
...
08/10/19 01:55:06 INFO mapred.Merger: Down to the last merge-pass, with 1
segments left of total size: 1111 bytes
08/10/19 01:55:06 WARN mapred.LocalJobRunner: job_local_0001
java.lang.NullPointerException
at test.TestWritable.readFields(TestWritable.java:36)
at
org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:67)
at
org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:40)
at
org.apache.hadoop.mapred.Task$ValuesIterator.readNextValue(Task.java:738)
..
On Sun, Oct 19, 2008 at 1:40 AM, Yih Sun Khoo <[EMAIL PROTECTED]> wrote:
> Also, I've noticed TupleWritable to be quite useful.
> What are good techniques for using TupleWritable in a mapping phase for a
> "list of Text" when you do not know the size of that "list" ahead of time
>
> Say I had a custom writable which implemented TupleWritable and the custom
> writable contained a setter method
> mycustomwritable.setTupleWritable( ... )
>
> Where the ellipsis is, there lies the TupleWritable. However I'm wondering
> since TupleWritable can be constructed using TupleWritable(Writable[]), how
> do I dynamically resize the Writable[] and add Text elements to it when I
> don't know the size of the Writable[] very well. Does this make sense?
>
>
>
> On Sun, Oct 19, 2008 at 1:32 AM, Yih Sun Khoo <[EMAIL PROTECTED]> wrote:
>
>> Let's say in the reduce phase your value happens to hold an
>> ArrayListWritable
>> In this example, value is of type ArrayListWritable
>> Maybe I've not thought about this or done this before, but how does one
>> "read data in from the DataInput stream" in the reduce phase so that the
>> ArrayListWritable which is a value already passed to the reducer can be used
>> as ArrayListWritable
>>
>>
>> On Sun, Oct 19, 2008 at 1:25 AM, Joman Chu <[EMAIL PROTECTED]> wrote:
>>
>>> Since the ArrayListWritable extends ArrayList, you have access to all
>>> the ArrayList methods as well. Once you read data in from the
>>> DataInput stream, you should be able to use ArrayListWritable just
>>> like a regular ArrayList.
>>> Joman Chu
>>> http://www.notatypewriter.com/
>>> AIM: ARcanUSNUMquam
>>>
>>>
>>>
>>> On Sun, Oct 19, 2008 at 4:16 AM, Yih Sun Khoo <[EMAIL PROTECTED]> wrote:
>>> > Hmm, what method from ArrayListWritable allows you to access the
>>> different
>>> > elements of the ArrayList? Would it be readFields? for example, in a
>>> > reduce phase, if I needed to know the size of the array list, it would
>>> be
>>> > easy if i were dealing with an arraylist because i could just say
>>> > arraylist.size. How would i accomplish that with the writable
>>> counterpart?
>>> >
>>> > On Sun, Oct 19, 2008 at 1:04 AM, Joman Chu <[EMAIL PROTECTED]> wrote:
>>> >
>>> >> Hi,
>>> >>
>>> >> For the ArrayList object, try taking a look at the implementation of
>>> >> ArrayListWritable by Jimmy Lin at UMD here:
>>> >>
>>> >>
>>> >>
>>> https://subversion.umiacs.umd.edu/umd-hadoop/core/trunk/src/edu/umd/cloud9/io/ArrayListWritable.java
>>> >>
>>> >> But basically in the readFields methods, I prefer using each Writable
>>> >> object's readFields method to read the data in. For example, for your
>>> >> double variable, I would use a DoubleWritable object and in the
>>> >> MyWritable.readFields(DataInput in), I would use
>>> >> nameofdoublewritable.readFields(in). For the
>>> >> MyWritable.write(DataOutput out) method, I would use
>>> >> nameofdoublewritable.write(out).
>>> >>
>>> >> Have a good one,
>>> >>
>>> >> Joman Chu
>>> >> http://www.notatypewriter.com/
>>> >> AIM: ARcanUSNUMquam
>>> >>
>>> >>
>>> >>
>>> >> On Sun, Oct 19, 2008 at 3:30 AM, Yih Sun Khoo <[EMAIL PROTECTED]>
>>> wrote:
>>> >> > I don't quite know how to write the read and write functions, but I
>>> want
>>> >> to
>>> >> > write my own writable, which should have a DoubleWritable/double
>>> value
>>> >> > followed by a list of Strings/Text. This Writable will be used as a
>>> >> value.
>>> >> > Is the code below the best way to go about writing such a writable?
>>> >> >
>>> >> > import java.io.DataInput;
>>> >> > import java.io.DataOutput;
>>> >> > import java.io.EOFException;
>>> >> > import java.io.IOException;
>>> >> > import java.util.ArrayList;
>>> >> >
>>> >> > import org.apache.hadoop.io.Writable;
>>> >> >
>>> >> > public class MyWritable implements Writable {
>>> >> > private double score;
>>> >> > private ArrayList<String> nameList;
>>> >> >
>>> >> > public void setScore(double score) {
>>> >> > this.score= score;
>>> >> > }
>>> >> >
>>> >> > public void setNameList(ArrayList<String> nameList) {
>>> >> > this.nameList= nameList;
>>> >> > }
>>> >> >
>>> >> > public double getScore() {
>>> >> > return score;
>>> >> > }
>>> >> >
>>> >> > public ArrayList<String> getNameList() {
>>> >> > return nameList;
>>> >> > }
>>> >> >
>>> >> > public void readFields(DataInput in) throws IOException {
>>> >> > score= in.readDouble();
>>> >> > try {
>>> >> > do {
>>> >> > nameList.add(in.readUTF());
>>> >> > } while (true);
>>> >> > } catch (EOFException eofe) {
>>> >> > // continue; done
>>> >> > }
>>> >> > }
>>> >> >
>>> >> > public void write(DataOutput out) throws IOException {
>>> >> > out.writeDouble(score);
>>> >> > for (String name: nameList) {
>>> >> > out.writeUTF(name);
>>> >> > }
>>> >> > }
>>> >> > }
>>> >> >
>>> >>
>>> >
>>>
>>
>>
>
