Re: How do I implement a Writable into another Writable?
Thanks Chris. Last question (pinkie promise) for this thread at least: public Iterator getUrlListAsArrayList(){ return urlList.iterator(); } is a better alternative to the clone() method because you had mentioned it will simply fail "in this case". Does "in this case" refer to when deep copy is used? On Mon, Oct 20, 2008 at 10:42 PM, Chris Douglas <[EMAIL PROTECTED]>wrote: > > On Oct 20, 2008, at 8:08 PM, Yih Sun Khoo wrote: > > Awesome! One question about the example you gave me. >> When you say "clears the collection", the expected value should just be >> "B0, >> B1" right? Because the collection gets cleared of the old contained value >> A0...A2. >> > > Sorry, I meant for the comments to refer to state *after* the execution of > the following line. So > > myStrings.addAll(foo.getNameList()); > > in the example is equivalent to > > myStrings.addAll(myStrings); > > since the reference stored in myStrings is the same collection returned > from getNameList(). The stmt will add the contents of the myString > collection ("B0", "B1") to itself. -C > > > MyWritable foo = new MyWritable(); >> >> // foo contains "A0", "A1", "A2" in its namelist >> foo.readFields(in); >> >> // attempt to save the collection of Strings >> Collection myStrings = foo.getNameList(); >> >> // clears the collection and replaces it with "B0", "B1" >> foo.readFields(in); >> >> // Expected: "A0", "A1", "A2", "B0", "B1" >> // Contains: "B0", "B1", "B0", "B1" >> myStrings.addAll(foo.getNameList()); >> >> >> On Mon, Oct 20, 2008 at 7:23 PM, Chris Douglas <[EMAIL PROTECTED] >> >wrote: >> >> >>> On Oct 20, 2008, at 6:43 PM, Yih Sun Khoo wrote: >>> >>> Thanks Chris and Joman for your detailed explanations. >>> Would this be a good example of using a shallow copy? Also I'm trying to wrap my head around why the shallow copy is needed. You mentioned it is to eliminate any state from the values the list might have formerly contained. Could you give me an example of when the deep copy implementation would have been terrible? >>> MyWritable foo = new MyWritable(); >>> >>> // foo contains "A0", "A1", "A2" in its namelist >>> foo.readFields(in); >>> >>> // attempt to save the collection of Strings >>> Collection myStrings = foo.getNameList(); >>> >>> // clears the collection and replaces it with "B0", "B1" >>> foo.readFields(in); >>> >>> // Expected: "A0", "A1", "A2", "B0", "B1" >>> // Contains: "B0", "B1", "B0", "B1" >>> myStrings.addAll(foo.getNameList()); >>> >>> >>> >>> @SuppressWarnings("unchecked") >>> *public* ArrayList getNameList() { *return* (ArrayList) nameList.clone(); } >>> You may not need the shallow copy, depending on how you're using the >>> type. >>> If your contract with the user is that getNameList() is invalid if >>> readFields is called again, you can call that out in documentation. The >>> more >>> canonical pattern is to return an iterator, which will fail in this case >>> (instead of silent, unexpected results). >>> >>> Also is it common to have instead used a one of the existing writables, >>> like DoubleWritable, as part of my custom writable. I would imagine in the readFields method I would call the DoubleWritable's readFields method. >>> There's usually no reason to allocate an object for a primitive type in a >>> composite type. It doesn't buy you much, since you're already using >>> DataInput and DataOutput interfaces, which define the wire format. >>> >>> What would go into (re)using a collection of Text instead of String? >>> Could >>> I perhaps use an ArrayList and in the readFields and write methods call each Text's readFields and write methods appropriately? >>> Depending on how your data looks, tracking the "valid" size of your >>> collection inside your type may permit you to reuse Text objects instead >>> of >>> creating new, immutable Strings. Unless you start noticing it as a >>> bottleneck, you're probably fine with Strings, though. -C >>> >>> >>> >>> On Mon, Oct 20, 2008 at 11:38 AM, Chris Douglas <[EMAIL PROTECTED] > wrote: > TupleWritable is not a general-purpose type. It's used for map-side > joins, > where the arity of a tuple is fixed by construction. Its intent is a > transient type with very, very specific applications in mind. > > It sounds like you don't need a general list type, as you don't need to > worry about encoding the type of object your list contains. Writables > are > *not* supposed to read to the end of the stream they're given; they are > to > consume a full instance from the stream (i.e. it must consume all "its" > bytes from a stream, even if it ultimately discards them). Given these > constraints, Writable types of variable size almost always encode their > length explicitly. As Joman mentioned, your constructor must initialize > all > it
Re: How do I implement a Writable into another Writable?
On Oct 20, 2008, at 8:08 PM, Yih Sun Khoo wrote: Awesome! One question about the example you gave me. When you say "clears the collection", the expected value should just be "B0, B1" right? Because the collection gets cleared of the old contained value A0...A2. Sorry, I meant for the comments to refer to state *after* the execution of the following line. So myStrings.addAll(foo.getNameList()); in the example is equivalent to myStrings.addAll(myStrings); since the reference stored in myStrings is the same collection returned from getNameList(). The stmt will add the contents of the myString collection ("B0", "B1") to itself. -C MyWritable foo = new MyWritable(); // foo contains "A0", "A1", "A2" in its namelist foo.readFields(in); // attempt to save the collection of Strings Collection myStrings = foo.getNameList(); // clears the collection and replaces it with "B0", "B1" foo.readFields(in); // Expected: "A0", "A1", "A2", "B0", "B1" // Contains: "B0", "B1", "B0", "B1" myStrings.addAll(foo.getNameList()); On Mon, Oct 20, 2008 at 7:23 PM, Chris Douglas <[EMAIL PROTECTED] inc.com>wrote: On Oct 20, 2008, at 6:43 PM, Yih Sun Khoo wrote: Thanks Chris and Joman for your detailed explanations. Would this be a good example of using a shallow copy? Also I'm trying to wrap my head around why the shallow copy is needed. You mentioned it is to eliminate any state from the values the list might have formerly contained. Could you give me an example of when the deep copy implementation would have been terrible? MyWritable foo = new MyWritable(); // foo contains "A0", "A1", "A2" in its namelist foo.readFields(in); // attempt to save the collection of Strings Collection myStrings = foo.getNameList(); // clears the collection and replaces it with "B0", "B1" foo.readFields(in); // Expected: "A0", "A1", "A2", "B0", "B1" // Contains: "B0", "B1", "B0", "B1" myStrings.addAll(foo.getNameList()); @SuppressWarnings("unchecked") *public* ArrayList getNameList() { *return* (ArrayList) nameList.clone(); } You may not need the shallow copy, depending on how you're using the type. If your contract with the user is that getNameList() is invalid if readFields is called again, you can call that out in documentation. The more canonical pattern is to return an iterator, which will fail in this case (instead of silent, unexpected results). Also is it common to have instead used a one of the existing writables, like DoubleWritable, as part of my custom writable. I would imagine in the readFields method I would call the DoubleWritable's readFields method. There's usually no reason to allocate an object for a primitive type in a composite type. It doesn't buy you much, since you're already using DataInput and DataOutput interfaces, which define the wire format. What would go into (re)using a collection of Text instead of String? Could I perhaps use an ArrayList and in the readFields and write methods call each Text's readFields and write methods appropriately? Depending on how your data looks, tracking the "valid" size of your collection inside your type may permit you to reuse Text objects instead of creating new, immutable Strings. Unless you start noticing it as a bottleneck, you're probably fine with Strings, though. -C On Mon, Oct 20, 2008 at 11:38 AM, Chris Douglas <[EMAIL PROTECTED] wrote: TupleWritable is not a general-purpose type. It's used for map-side joins, where the arity of a tuple is fixed by construction. Its intent is a transient type with very, very specific applications in mind. It sounds like you don't need a general list type, as you don't need to worry about encoding the type of object your list contains. Writables are *not* supposed to read to the end of the stream they're given; they are to consume a full instance from the stream (i.e. it must consume all "its" bytes from a stream, even if it ultimately discards them). Given these constraints, Writable types of variable size almost always encode their length explicitly. As Joman mentioned, your constructor must initialize all its elements. Further, readFields must not retain any state from the value it formerly contained, so you need to clear the list before you add more values to it. This means your getNameList method will need to do a shallow copy of its elements if the caller stores a reference to the list. This should work: public void readFields(DataInput in) throws IOException { nameList.clear(); score = in.readDouble(); final int len = WritableUtils.readVInt(in); for (int i = 0; i < len; ++i) { nameList.add(Text.readString(in)); } } public void write(DataOutput out) throws IOException { out.writeDouble(score); WritableUtils.writeVInt(out, nameList.size()); for (String name : nameList) { Text.writeString(out, name); } } You can improve your performance by (re)using a collection of Text instead of String (since the latter is immu
Re: How do I implement a Writable into another Writable?
Awesome! One question about the example you gave me. When you say "clears the collection", the expected value should just be "B0, B1" right? Because the collection gets cleared of the old contained value A0...A2. MyWritable foo = new MyWritable(); // foo contains "A0", "A1", "A2" in its namelist foo.readFields(in); // attempt to save the collection of Strings Collection myStrings = foo.getNameList(); // clears the collection and replaces it with "B0", "B1" foo.readFields(in); // Expected: "A0", "A1", "A2", "B0", "B1" // Contains: "B0", "B1", "B0", "B1" myStrings.addAll(foo.getNameList()); On Mon, Oct 20, 2008 at 7:23 PM, Chris Douglas <[EMAIL PROTECTED]>wrote: > > On Oct 20, 2008, at 6:43 PM, Yih Sun Khoo wrote: > > Thanks Chris and Joman for your detailed explanations. >> >> Would this be a good example of using a shallow copy? Also I'm trying to >> wrap my head around why the shallow copy is needed. You mentioned it is to >> eliminate any state from the values the list might have formerly >> contained. >> Could you give me an example of when the deep copy implementation would >> have >> been terrible? >> > > MyWritable foo = new MyWritable(); > > // foo contains "A0", "A1", "A2" in its namelist > foo.readFields(in); > > // attempt to save the collection of Strings > Collection myStrings = foo.getNameList(); > > // clears the collection and replaces it with "B0", "B1" > foo.readFields(in); > > // Expected: "A0", "A1", "A2", "B0", "B1" > // Contains: "B0", "B1", "B0", "B1" > myStrings.addAll(foo.getNameList()); > > > > @SuppressWarnings("unchecked") >> >> *public* ArrayList getNameList() { >> >> *return* (ArrayList) nameList.clone(); >> >> } >> > > You may not need the shallow copy, depending on how you're using the type. > If your contract with the user is that getNameList() is invalid if > readFields is called again, you can call that out in documentation. The more > canonical pattern is to return an iterator, which will fail in this case > (instead of silent, unexpected results). > > Also is it common to have instead used a one of the existing writables, >> like >> DoubleWritable, as part of my custom writable. I would imagine in the >> readFields method I would call the DoubleWritable's readFields method. >> > > There's usually no reason to allocate an object for a primitive type in a > composite type. It doesn't buy you much, since you're already using > DataInput and DataOutput interfaces, which define the wire format. > > What would go into (re)using a collection of Text instead of String? Could >> I perhaps use an ArrayList and in the readFields and write methods >> call each Text's readFields and write methods appropriately? >> > > Depending on how your data looks, tracking the "valid" size of your > collection inside your type may permit you to reuse Text objects instead of > creating new, immutable Strings. Unless you start noticing it as a > bottleneck, you're probably fine with Strings, though. -C > > > >> On Mon, Oct 20, 2008 at 11:38 AM, Chris Douglas <[EMAIL PROTECTED] >> >wrote: >> >> TupleWritable is not a general-purpose type. It's used for map-side >>> joins, >>> where the arity of a tuple is fixed by construction. Its intent is a >>> transient type with very, very specific applications in mind. >>> >>> It sounds like you don't need a general list type, as you don't need to >>> worry about encoding the type of object your list contains. Writables are >>> *not* supposed to read to the end of the stream they're given; they are >>> to >>> consume a full instance from the stream (i.e. it must consume all "its" >>> bytes from a stream, even if it ultimately discards them). Given these >>> constraints, Writable types of variable size almost always encode their >>> length explicitly. As Joman mentioned, your constructor must initialize >>> all >>> its elements. Further, readFields must not retain any state from the >>> value >>> it formerly contained, so you need to clear the list before you add more >>> values to it. This means your getNameList method will need to do a >>> shallow >>> copy of its elements if the caller stores a reference to the list. >>> >>> This should work: >>> >>> public void readFields(DataInput in) throws IOException { >>> nameList.clear(); >>> score = in.readDouble(); >>> final int len = WritableUtils.readVInt(in); >>> for (int i = 0; i < len; ++i) { >>>nameList.add(Text.readString(in)); >>> } >>> } >>> >>> public void write(DataOutput out) throws IOException { >>> out.writeDouble(score); >>> WritableUtils.writeVInt(out, nameList.size()); >>> for (String name : nameList) { >>>Text.writeString(out, name); >>> } >>> } >>> >>> You can improve your performance by (re)using a collection of Text >>> instead >>> of String (since the latter is immutable), but that requires more work. >>> -C >>> >>> >>> On Oct 19, 2008, at 3:39 PM, Yih Sun Khoo wrote: >>> >>> I think when it comes to the TupleWritable being part of a custm >>> writable, >>> y
Re: How do I implement a Writable into another Writable?
On Oct 20, 2008, at 6:43 PM, Yih Sun Khoo wrote: Thanks Chris and Joman for your detailed explanations. Would this be a good example of using a shallow copy? Also I'm trying to wrap my head around why the shallow copy is needed. You mentioned it is to eliminate any state from the values the list might have formerly contained. Could you give me an example of when the deep copy implementation would have been terrible? MyWritable foo = new MyWritable(); // foo contains "A0", "A1", "A2" in its namelist foo.readFields(in); // attempt to save the collection of Strings Collection myStrings = foo.getNameList(); // clears the collection and replaces it with "B0", "B1" foo.readFields(in); // Expected: "A0", "A1", "A2", "B0", "B1" // Contains: "B0", "B1", "B0", "B1" myStrings.addAll(foo.getNameList()); @SuppressWarnings("unchecked") *public* ArrayList getNameList() { *return* (ArrayList) nameList.clone(); } You may not need the shallow copy, depending on how you're using the type. If your contract with the user is that getNameList() is invalid if readFields is called again, you can call that out in documentation. The more canonical pattern is to return an iterator, which will fail in this case (instead of silent, unexpected results). Also is it common to have instead used a one of the existing writables, like DoubleWritable, as part of my custom writable. I would imagine in the readFields method I would call the DoubleWritable's readFields method. There's usually no reason to allocate an object for a primitive type in a composite type. It doesn't buy you much, since you're already using DataInput and DataOutput interfaces, which define the wire format. What would go into (re)using a collection of Text instead of String? Could I perhaps use an ArrayList and in the readFields and write methods call each Text's readFields and write methods appropriately? Depending on how your data looks, tracking the "valid" size of your collection inside your type may permit you to reuse Text objects instead of creating new, immutable Strings. Unless you start noticing it as a bottleneck, you're probably fine with Strings, though. -C On Mon, Oct 20, 2008 at 11:38 AM, Chris Douglas <[EMAIL PROTECTED] inc.com>wrote: TupleWritable is not a general-purpose type. It's used for map-side joins, where the arity of a tuple is fixed by construction. Its intent is a transient type with very, very specific applications in mind. It sounds like you don't need a general list type, as you don't need to worry about encoding the type of object your list contains. Writables are *not* supposed to read to the end of the stream they're given; they are to consume a full instance from the stream (i.e. it must consume all "its" bytes from a stream, even if it ultimately discards them). Given these constraints, Writable types of variable size almost always encode their length explicitly. As Joman mentioned, your constructor must initialize all its elements. Further, readFields must not retain any state from the value it formerly contained, so you need to clear the list before you add more values to it. This means your getNameList method will need to do a shallow copy of its elements if the caller stores a reference to the list. This should work: public void readFields(DataInput in) throws IOException { nameList.clear(); score = in.readDouble(); final int len = WritableUtils.readVInt(in); for (int i = 0; i < len; ++i) { nameList.add(Text.readString(in)); } } public void write(DataOutput out) throws IOException { out.writeDouble(score); WritableUtils.writeVInt(out, nameList.size()); for (String name : nameList) { Text.writeString(out, name); } } You can improve your performance by (re)using a collection of Text instead of String (since the latter is immutable), but that requires more work. -C On Oct 19, 2008, at 3:39 PM, Yih Sun Khoo wrote: I think when it comes to the TupleWritable being part of a custm writable, you cannot just say tupleWritable.readFields(in) and tupleWritable.write(out) I might be wrong. Has anyone successfully implemented a TupleWritable with ,say, a DoubleWritable in a custom writable? On Sun, Oct 19, 2008 at 3:33 AM, Joman Chu <[EMAIL PROTECTED]> wrote: hrm, try implementing the read(DataInput in) method, as well as a blank constructor MyWritable() that fills dummy values into your instance variables. For example this should be all you need for read(DataInput in), public static MyWritable read(DataInput in) throws IOException { MyWritable w = new MyWritable(); w.readFields(in); return w; } EDIT: I was able to sort of replicate your error. In my constructor, i had my instance variables assigned to null. Make sure you assign them to new instances of whatever Writable you are using. Joman Chu http://www.notatypewriter.com/ AIM: ARcanUSNUMquam On Sun, Oct 19, 2008 at 5:10 AM,
Re: How do I implement a Writable into another Writable?
Thanks Chris and Joman for your detailed explanations. Would this be a good example of using a shallow copy? Also I'm trying to wrap my head around why the shallow copy is needed. You mentioned it is to eliminate any state from the values the list might have formerly contained. Could you give me an example of when the deep copy implementation would have been terrible? @SuppressWarnings("unchecked") *public* ArrayList getNameList() { *return* (ArrayList) nameList.clone(); } Also is it common to have instead used a one of the existing writables, like DoubleWritable, as part of my custom writable. I would imagine in the readFields method I would call the DoubleWritable's readFields method. What would go into (re)using a collection of Text instead of String? Could I perhaps use an ArrayList and in the readFields and write methods call each Text's readFields and write methods appropriately? On Mon, Oct 20, 2008 at 11:38 AM, Chris Douglas <[EMAIL PROTECTED]>wrote: > TupleWritable is not a general-purpose type. It's used for map-side joins, > where the arity of a tuple is fixed by construction. Its intent is a > transient type with very, very specific applications in mind. > > It sounds like you don't need a general list type, as you don't need to > worry about encoding the type of object your list contains. Writables are > *not* supposed to read to the end of the stream they're given; they are to > consume a full instance from the stream (i.e. it must consume all "its" > bytes from a stream, even if it ultimately discards them). Given these > constraints, Writable types of variable size almost always encode their > length explicitly. As Joman mentioned, your constructor must initialize all > its elements. Further, readFields must not retain any state from the value > it formerly contained, so you need to clear the list before you add more > values to it. This means your getNameList method will need to do a shallow > copy of its elements if the caller stores a reference to the list. > > This should work: > > public void readFields(DataInput in) throws IOException { >nameList.clear(); >score = in.readDouble(); >final int len = WritableUtils.readVInt(in); >for (int i = 0; i < len; ++i) { > nameList.add(Text.readString(in)); >} > } > > public void write(DataOutput out) throws IOException { >out.writeDouble(score); >WritableUtils.writeVInt(out, nameList.size()); >for (String name : nameList) { > Text.writeString(out, name); >} > } > > You can improve your performance by (re)using a collection of Text instead > of String (since the latter is immutable), but that requires more work. -C > > > On Oct 19, 2008, at 3:39 PM, Yih Sun Khoo wrote: > > I think when it comes to the TupleWritable being part of a custm writable, >> you cannot just say tupleWritable.readFields(in) and >> tupleWritable.write(out) >> >> I might be wrong. Has anyone successfully implemented a TupleWritable >> with >> ,say, a DoubleWritable in a custom writable? >> >> On Sun, Oct 19, 2008 at 3:33 AM, Joman Chu <[EMAIL PROTECTED]> wrote: >> >> hrm, try implementing the read(DataInput in) method, as well as a >>> blank constructor MyWritable() that fills dummy values into your >>> instance variables. For example this should be all you need for >>> read(DataInput in), >>> >>> public static MyWritable read(DataInput in) throws IOException { >>> MyWritable w = new MyWritable(); >>> w.readFields(in); >>> return w; >>> } >>> >>> EDIT: I was able to sort of replicate your error. In my constructor, i >>> had my instance variables assigned to null. Make sure you assign them >>> to new instances of whatever Writable you are using. >>> >>> >>> Joman Chu >>> http://www.notatypewriter.com/ >>> AIM: ARcanUSNUMquam >>> >>> >>> >>> On Sun, Oct 19, 2008 at 5:10 AM, Yih Sun Khoo <[EMAIL PROTECTED]> wrote: >>> Joman to add a little bit more to one of my previous mails about the readFields methods Have you ever had something like this? public class MyWritable implements Writable { private DoubleWritable doubleWritable; private TupleWritable tupleWritable; public void readFields(DataInput in) throws IOException { doubleWritable.readFields(in); tupleWritable.readFields(in); } public void write(DataOutput out) throws IOException { doubleWritable.write(out); tupleWritable.write(out); } } On Sun, Oct 19, 2008 at 1:59 AM, Joman Chu <[EMAIL PROTECTED]> wrote: I've never used TupleWritable, so hopefully somebody else can help you > with that. > Joman Chu > http://www.notatypewriter.com/ > AIM: ARcanUSNUMquam > > > > On Sun, Oct 19, 2008 at 4:40 AM, Yih Sun Khoo <[EMAIL PROTECTED]> > wrote: > >> Also, I've noticed TupleWritable to be quite useful. >> What are good techniques for using TupleWritable in a ma
Re: How do I implement a Writable into another Writable?
TupleWritable is not a general-purpose type. It's used for map-side joins, where the arity of a tuple is fixed by construction. Its intent is a transient type with very, very specific applications in mind. It sounds like you don't need a general list type, as you don't need to worry about encoding the type of object your list contains. Writables are *not* supposed to read to the end of the stream they're given; they are to consume a full instance from the stream (i.e. it must consume all "its" bytes from a stream, even if it ultimately discards them). Given these constraints, Writable types of variable size almost always encode their length explicitly. As Joman mentioned, your constructor must initialize all its elements. Further, readFields must not retain any state from the value it formerly contained, so you need to clear the list before you add more values to it. This means your getNameList method will need to do a shallow copy of its elements if the caller stores a reference to the list. This should work: public void readFields(DataInput in) throws IOException { nameList.clear(); score = in.readDouble(); final int len = WritableUtils.readVInt(in); for (int i = 0; i < len; ++i) { nameList.add(Text.readString(in)); } } public void write(DataOutput out) throws IOException { out.writeDouble(score); WritableUtils.writeVInt(out, nameList.size()); for (String name : nameList) { Text.writeString(out, name); } } You can improve your performance by (re)using a collection of Text instead of String (since the latter is immutable), but that requires more work. -C On Oct 19, 2008, at 3:39 PM, Yih Sun Khoo wrote: I think when it comes to the TupleWritable being part of a custm writable, you cannot just say tupleWritable.readFields(in) and tupleWritable.write(out) I might be wrong. Has anyone successfully implemented a TupleWritable with ,say, a DoubleWritable in a custom writable? On Sun, Oct 19, 2008 at 3:33 AM, Joman Chu <[EMAIL PROTECTED]> wrote: hrm, try implementing the read(DataInput in) method, as well as a blank constructor MyWritable() that fills dummy values into your instance variables. For example this should be all you need for read(DataInput in), public static MyWritable read(DataInput in) throws IOException { MyWritable w = new MyWritable(); w.readFields(in); return w; } EDIT: I was able to sort of replicate your error. In my constructor, i had my instance variables assigned to null. Make sure you assign them to new instances of whatever Writable you are using. Joman Chu http://www.notatypewriter.com/ AIM: ARcanUSNUMquam On Sun, Oct 19, 2008 at 5:10 AM, Yih Sun Khoo <[EMAIL PROTECTED]> wrote: Joman to add a little bit more to one of my previous mails about the readFields methods Have you ever had something like this? public class MyWritable implements Writable { private DoubleWritable doubleWritable; private TupleWritable tupleWritable; public void readFields(DataInput in) throws IOException { doubleWritable.readFields(in); tupleWritable.readFields(in); } public void write(DataOutput out) throws IOException { doubleWritable.write(out); tupleWritable.write(out); } } On Sun, Oct 19, 2008 at 1:59 AM, Joman Chu <[EMAIL PROTECTED]> wrote: I've never used TupleWritable, so hopefully somebody else can help you with that. Joman Chu http://www.notatypewriter.com/ AIM: ARcanUSNUMquam On Sun, Oct 19, 2008 at 4:40 AM, Yih Sun Khoo <[EMAIL PROTECTED]> wrote: Also, I've noticed TupleWritable to be quite useful. What are good techniques for using TupleWritable in a mapping phase for a "list of Text" when you do not know the size of that "list" ahead of time Say I had a custom writable which implemented TupleWritable and the custom writable contained a setter method mycustomwritable.setTupleWritable( ... ) Where the ellipsis is, there lies the TupleWritable. However I'm wondering since TupleWritable can be constructed using TupleWritable(Writable[]), how do I dynamically resize the Writable[] and add Text elements to it when I don't know the size of the Writable[] very well. Does this make sense? On Sun, Oct 19, 2008 at 1:32 AM, Yih Sun Khoo <[EMAIL PROTECTED]> wrote: Let's say in the reduce phase your value happens to hold an ArrayListWritable In this example, value is of type ArrayListWritable Maybe I've not thought about this or done this before, but how does one "read data in from the DataInput stream" in the reduce phase so that the ArrayListWritable which is a value already passed to the reducer can be used as ArrayListWritable On Sun, Oct 19, 2008 at 1:25 AM, Joman Chu <[EMAIL PROTECTED]> wrote: Since the ArrayListWritable extends ArrayList, you have access to all the ArrayList methods as well. Once you read data in from the DataInput stream, you should be able to use A
Re: How do I implement a Writable into another Writable?
I think when it comes to the TupleWritable being part of a custm writable, you cannot just say tupleWritable.readFields(in) and tupleWritable.write(out) I might be wrong. Has anyone successfully implemented a TupleWritable with ,say, a DoubleWritable in a custom writable? On Sun, Oct 19, 2008 at 3:33 AM, Joman Chu <[EMAIL PROTECTED]> wrote: > hrm, try implementing the read(DataInput in) method, as well as a > blank constructor MyWritable() that fills dummy values into your > instance variables. For example this should be all you need for > read(DataInput in), > > public static MyWritable read(DataInput in) throws IOException { >MyWritable w = new MyWritable(); >w.readFields(in); >return w; > } > > EDIT: I was able to sort of replicate your error. In my constructor, i > had my instance variables assigned to null. Make sure you assign them > to new instances of whatever Writable you are using. > > > Joman Chu > http://www.notatypewriter.com/ > AIM: ARcanUSNUMquam > > > > On Sun, Oct 19, 2008 at 5:10 AM, Yih Sun Khoo <[EMAIL PROTECTED]> wrote: > > Joman to add a little bit more to one of my previous mails about the > > readFields methods > > > > Have you ever had something like this? > > > > public class MyWritable implements Writable { > >private DoubleWritable doubleWritable; > >private TupleWritable tupleWritable; > > > >public void readFields(DataInput in) throws IOException { > >doubleWritable.readFields(in); > >tupleWritable.readFields(in); > >} > > > >public void write(DataOutput out) throws IOException { > >doubleWritable.write(out); > >tupleWritable.write(out); > >} > > > > > > } > > > > On Sun, Oct 19, 2008 at 1:59 AM, Joman Chu <[EMAIL PROTECTED]> wrote: > > > >> I've never used TupleWritable, so hopefully somebody else can help you > >> with that. > >> Joman Chu > >> http://www.notatypewriter.com/ > >> AIM: ARcanUSNUMquam > >> > >> > >> > >> On Sun, Oct 19, 2008 at 4:40 AM, Yih Sun Khoo <[EMAIL PROTECTED]> wrote: > >> > Also, I've noticed TupleWritable to be quite useful. > >> > What are good techniques for using TupleWritable in a mapping phase > for a > >> > "list of Text" when you do not know the size of that "list" ahead of > time > >> > > >> > Say I had a custom writable which implemented TupleWritable and the > >> custom > >> > writable contained a setter method > >> > mycustomwritable.setTupleWritable( ... ) > >> > > >> > Where the ellipsis is, there lies the TupleWritable. However I'm > >> wondering > >> > since TupleWritable can be constructed using > TupleWritable(Writable[]), > >> how > >> > do I dynamically resize the Writable[] and add Text elements to it > when I > >> > don't know the size of the Writable[] very well. Does this make > sense? > >> > > >> > > >> > On Sun, Oct 19, 2008 at 1:32 AM, Yih Sun Khoo <[EMAIL PROTECTED]> > wrote: > >> > > >> >> Let's say in the reduce phase your value happens to hold an > >> >> ArrayListWritable > >> >> In this example, value is of type ArrayListWritable > >> >> Maybe I've not thought about this or done this before, but how does > one > >> >> "read data in from the DataInput stream" in the reduce phase so that > the > >> >> ArrayListWritable which is a value already passed to the reducer can > be > >> used > >> >> as ArrayListWritable > >> >> > >> >> > >> >> On Sun, Oct 19, 2008 at 1:25 AM, Joman Chu <[EMAIL PROTECTED]> > wrote: > >> >> > >> >>> Since the ArrayListWritable extends ArrayList, you have access to > all > >> >>> the ArrayList methods as well. Once you read data in from the > >> >>> DataInput stream, you should be able to use ArrayListWritable just > >> >>> like a regular ArrayList. > >> >>> Joman Chu > >> >>> http://www.notatypewriter.com/ > >> >>> AIM: ARcanUSNUMquam > >> >>> > >> >>> > >> >>> > >> >>> On Sun, Oct 19, 2008 at 4:16 AM, Yih Sun Khoo <[EMAIL PROTECTED]> > >> wrote: > >> >>> > Hmm, what method from ArrayListWritable allows you to access the > >> >>> different > >> >>> > elements of the ArrayList? Would it be readFields? for example, > in > >> a > >> >>> > reduce phase, if I needed to know the size of the array list, it > >> would > >> >>> be > >> >>> > easy if i were dealing with an arraylist because i could just say > >> >>> > arraylist.size. How would i accomplish that with the writable > >> >>> counterpart? > >> >>> > > >> >>> > On Sun, Oct 19, 2008 at 1:04 AM, Joman Chu <[EMAIL PROTECTED]> > >> wrote: > >> >>> > > >> >>> >> Hi, > >> >>> >> > >> >>> >> For the ArrayList object, try taking a look at the implementation > of > >> >>> >> ArrayListWritable by Jimmy Lin at UMD here: > >> >>> >> > >> >>> >> > >> >>> >> > >> >>> > >> > https://subversion.umiacs.umd.edu/umd-hadoop/core/trunk/src/edu/umd/cloud9/io/ArrayListWritable.java > >> >>> >> > >> >>> >> But basically in the readFields methods, I prefer using each > >> Writable > >> >>> >> object's readFields method to read the data in. For example, for > >> your > >> >>> >> doubl
Re: How do I implement a Writable into another Writable?
hrm, try implementing the read(DataInput in) method, as well as a blank constructor MyWritable() that fills dummy values into your instance variables. For example this should be all you need for read(DataInput in), public static MyWritable read(DataInput in) throws IOException { MyWritable w = new MyWritable(); w.readFields(in); return w; } EDIT: I was able to sort of replicate your error. In my constructor, i had my instance variables assigned to null. Make sure you assign them to new instances of whatever Writable you are using. Joman Chu http://www.notatypewriter.com/ AIM: ARcanUSNUMquam On Sun, Oct 19, 2008 at 5:10 AM, Yih Sun Khoo <[EMAIL PROTECTED]> wrote: > Joman to add a little bit more to one of my previous mails about the > readFields methods > > Have you ever had something like this? > > public class MyWritable implements Writable { >private DoubleWritable doubleWritable; >private TupleWritable tupleWritable; > >public void readFields(DataInput in) throws IOException { >doubleWritable.readFields(in); >tupleWritable.readFields(in); >} > >public void write(DataOutput out) throws IOException { >doubleWritable.write(out); >tupleWritable.write(out); >} > > > } > > On Sun, Oct 19, 2008 at 1:59 AM, Joman Chu <[EMAIL PROTECTED]> wrote: > >> I've never used TupleWritable, so hopefully somebody else can help you >> with that. >> Joman Chu >> http://www.notatypewriter.com/ >> AIM: ARcanUSNUMquam >> >> >> >> On Sun, Oct 19, 2008 at 4:40 AM, Yih Sun Khoo <[EMAIL PROTECTED]> wrote: >> > Also, I've noticed TupleWritable to be quite useful. >> > What are good techniques for using TupleWritable in a mapping phase for a >> > "list of Text" when you do not know the size of that "list" ahead of time >> > >> > Say I had a custom writable which implemented TupleWritable and the >> custom >> > writable contained a setter method >> > mycustomwritable.setTupleWritable( ... ) >> > >> > Where the ellipsis is, there lies the TupleWritable. However I'm >> wondering >> > since TupleWritable can be constructed using TupleWritable(Writable[]), >> how >> > do I dynamically resize the Writable[] and add Text elements to it when I >> > don't know the size of the Writable[] very well. Does this make sense? >> > >> > >> > On Sun, Oct 19, 2008 at 1:32 AM, Yih Sun Khoo <[EMAIL PROTECTED]> wrote: >> > >> >> Let's say in the reduce phase your value happens to hold an >> >> ArrayListWritable >> >> In this example, value is of type ArrayListWritable >> >> Maybe I've not thought about this or done this before, but how does one >> >> "read data in from the DataInput stream" in the reduce phase so that the >> >> ArrayListWritable which is a value already passed to the reducer can be >> used >> >> as ArrayListWritable >> >> >> >> >> >> On Sun, Oct 19, 2008 at 1:25 AM, Joman Chu <[EMAIL PROTECTED]> wrote: >> >> >> >>> Since the ArrayListWritable extends ArrayList, you have access to all >> >>> the ArrayList methods as well. Once you read data in from the >> >>> DataInput stream, you should be able to use ArrayListWritable just >> >>> like a regular ArrayList. >> >>> Joman Chu >> >>> http://www.notatypewriter.com/ >> >>> AIM: ARcanUSNUMquam >> >>> >> >>> >> >>> >> >>> On Sun, Oct 19, 2008 at 4:16 AM, Yih Sun Khoo <[EMAIL PROTECTED]> >> wrote: >> >>> > Hmm, what method from ArrayListWritable allows you to access the >> >>> different >> >>> > elements of the ArrayList? Would it be readFields? for example, in >> a >> >>> > reduce phase, if I needed to know the size of the array list, it >> would >> >>> be >> >>> > easy if i were dealing with an arraylist because i could just say >> >>> > arraylist.size. How would i accomplish that with the writable >> >>> counterpart? >> >>> > >> >>> > On Sun, Oct 19, 2008 at 1:04 AM, Joman Chu <[EMAIL PROTECTED]> >> wrote: >> >>> > >> >>> >> Hi, >> >>> >> >> >>> >> For the ArrayList object, try taking a look at the implementation of >> >>> >> ArrayListWritable by Jimmy Lin at UMD here: >> >>> >> >> >>> >> >> >>> >> >> >>> >> https://subversion.umiacs.umd.edu/umd-hadoop/core/trunk/src/edu/umd/cloud9/io/ArrayListWritable.java >> >>> >> >> >>> >> But basically in the readFields methods, I prefer using each >> Writable >> >>> >> object's readFields method to read the data in. For example, for >> your >> >>> >> double variable, I would use a DoubleWritable object and in the >> >>> >> MyWritable.readFields(DataInput in), I would use >> >>> >> nameofdoublewritable.readFields(in). For the >> >>> >> MyWritable.write(DataOutput out) method, I would use >> >>> >> nameofdoublewritable.write(out). >> >>> >> >> >>> >> Have a good one, >> >>> >> >> >>> >> Joman Chu >> >>> >> http://www.notatypewriter.com/ >> >>> >> AIM: ARcanUSNUMquam >> >>> >> >> >>> >> >> >>> >> >> >>> >> On Sun, Oct 19, 2008 at 3:30 AM, Yih Sun Khoo <[EMAIL PROTECTED]> >> >>> wrote: >> >>> >> > I don't quite know how to write the read and write functions, but >> I >
Re: How do I implement a Writable into another Writable?
Joman to add a little bit more to one of my previous mails about the readFields methods Have you ever had something like this? public class MyWritable implements Writable { private DoubleWritable doubleWritable; private TupleWritable tupleWritable; public void readFields(DataInput in) throws IOException { doubleWritable.readFields(in); tupleWritable.readFields(in); } public void write(DataOutput out) throws IOException { doubleWritable.write(out); tupleWritable.write(out); } } On Sun, Oct 19, 2008 at 1:59 AM, Joman Chu <[EMAIL PROTECTED]> wrote: > I've never used TupleWritable, so hopefully somebody else can help you > with that. > Joman Chu > http://www.notatypewriter.com/ > AIM: ARcanUSNUMquam > > > > On Sun, Oct 19, 2008 at 4:40 AM, Yih Sun Khoo <[EMAIL PROTECTED]> wrote: > > Also, I've noticed TupleWritable to be quite useful. > > What are good techniques for using TupleWritable in a mapping phase for a > > "list of Text" when you do not know the size of that "list" ahead of time > > > > Say I had a custom writable which implemented TupleWritable and the > custom > > writable contained a setter method > > mycustomwritable.setTupleWritable( ... ) > > > > Where the ellipsis is, there lies the TupleWritable. However I'm > wondering > > since TupleWritable can be constructed using TupleWritable(Writable[]), > how > > do I dynamically resize the Writable[] and add Text elements to it when I > > don't know the size of the Writable[] very well. Does this make sense? > > > > > > On Sun, Oct 19, 2008 at 1:32 AM, Yih Sun Khoo <[EMAIL PROTECTED]> wrote: > > > >> Let's say in the reduce phase your value happens to hold an > >> ArrayListWritable > >> In this example, value is of type ArrayListWritable > >> Maybe I've not thought about this or done this before, but how does one > >> "read data in from the DataInput stream" in the reduce phase so that the > >> ArrayListWritable which is a value already passed to the reducer can be > used > >> as ArrayListWritable > >> > >> > >> On Sun, Oct 19, 2008 at 1:25 AM, Joman Chu <[EMAIL PROTECTED]> wrote: > >> > >>> Since the ArrayListWritable extends ArrayList, you have access to all > >>> the ArrayList methods as well. Once you read data in from the > >>> DataInput stream, you should be able to use ArrayListWritable just > >>> like a regular ArrayList. > >>> Joman Chu > >>> http://www.notatypewriter.com/ > >>> AIM: ARcanUSNUMquam > >>> > >>> > >>> > >>> On Sun, Oct 19, 2008 at 4:16 AM, Yih Sun Khoo <[EMAIL PROTECTED]> > wrote: > >>> > Hmm, what method from ArrayListWritable allows you to access the > >>> different > >>> > elements of the ArrayList? Would it be readFields? for example, in > a > >>> > reduce phase, if I needed to know the size of the array list, it > would > >>> be > >>> > easy if i were dealing with an arraylist because i could just say > >>> > arraylist.size. How would i accomplish that with the writable > >>> counterpart? > >>> > > >>> > On Sun, Oct 19, 2008 at 1:04 AM, Joman Chu <[EMAIL PROTECTED]> > wrote: > >>> > > >>> >> Hi, > >>> >> > >>> >> For the ArrayList object, try taking a look at the implementation of > >>> >> ArrayListWritable by Jimmy Lin at UMD here: > >>> >> > >>> >> > >>> >> > >>> > https://subversion.umiacs.umd.edu/umd-hadoop/core/trunk/src/edu/umd/cloud9/io/ArrayListWritable.java > >>> >> > >>> >> But basically in the readFields methods, I prefer using each > Writable > >>> >> object's readFields method to read the data in. For example, for > your > >>> >> double variable, I would use a DoubleWritable object and in the > >>> >> MyWritable.readFields(DataInput in), I would use > >>> >> nameofdoublewritable.readFields(in). For the > >>> >> MyWritable.write(DataOutput out) method, I would use > >>> >> nameofdoublewritable.write(out). > >>> >> > >>> >> Have a good one, > >>> >> > >>> >> Joman Chu > >>> >> http://www.notatypewriter.com/ > >>> >> AIM: ARcanUSNUMquam > >>> >> > >>> >> > >>> >> > >>> >> On Sun, Oct 19, 2008 at 3:30 AM, Yih Sun Khoo <[EMAIL PROTECTED]> > >>> wrote: > >>> >> > I don't quite know how to write the read and write functions, but > I > >>> want > >>> >> to > >>> >> > write my own writable, which should have a DoubleWritable/double > >>> value > >>> >> > followed by a list of Strings/Text. This Writable will be used as > a > >>> >> value. > >>> >> > Is the code below the best way to go about writing such a > writable? > >>> >> > > >>> >> > import java.io.DataInput; > >>> >> > import java.io.DataOutput; > >>> >> > import java.io.EOFException; > >>> >> > import java.io.IOException; > >>> >> > import java.util.ArrayList; > >>> >> > > >>> >> > import org.apache.hadoop.io.Writable; > >>> >> > > >>> >> > public class MyWritable implements Writable { > >>> >> >private double score; > >>> >> >private ArrayList nameList; > >>> >> > > >>> >> >public void setScore(double score) { > >>> >> >this.score= score; > >>> >> >} > >>> >> > > >>>
Re: How do I implement a Writable into another Writable?
I've never used TupleWritable, so hopefully somebody else can help you with that. Joman Chu http://www.notatypewriter.com/ AIM: ARcanUSNUMquam On Sun, Oct 19, 2008 at 4:40 AM, Yih Sun Khoo <[EMAIL PROTECTED]> wrote: > Also, I've noticed TupleWritable to be quite useful. > What are good techniques for using TupleWritable in a mapping phase for a > "list of Text" when you do not know the size of that "list" ahead of time > > Say I had a custom writable which implemented TupleWritable and the custom > writable contained a setter method > mycustomwritable.setTupleWritable( ... ) > > Where the ellipsis is, there lies the TupleWritable. However I'm wondering > since TupleWritable can be constructed using TupleWritable(Writable[]), how > do I dynamically resize the Writable[] and add Text elements to it when I > don't know the size of the Writable[] very well. Does this make sense? > > > On Sun, Oct 19, 2008 at 1:32 AM, Yih Sun Khoo <[EMAIL PROTECTED]> wrote: > >> Let's say in the reduce phase your value happens to hold an >> ArrayListWritable >> In this example, value is of type ArrayListWritable >> Maybe I've not thought about this or done this before, but how does one >> "read data in from the DataInput stream" in the reduce phase so that the >> ArrayListWritable which is a value already passed to the reducer can be used >> as ArrayListWritable >> >> >> On Sun, Oct 19, 2008 at 1:25 AM, Joman Chu <[EMAIL PROTECTED]> wrote: >> >>> Since the ArrayListWritable extends ArrayList, you have access to all >>> the ArrayList methods as well. Once you read data in from the >>> DataInput stream, you should be able to use ArrayListWritable just >>> like a regular ArrayList. >>> Joman Chu >>> http://www.notatypewriter.com/ >>> AIM: ARcanUSNUMquam >>> >>> >>> >>> On Sun, Oct 19, 2008 at 4:16 AM, Yih Sun Khoo <[EMAIL PROTECTED]> wrote: >>> > Hmm, what method from ArrayListWritable allows you to access the >>> different >>> > elements of the ArrayList? Would it be readFields? for example, in a >>> > reduce phase, if I needed to know the size of the array list, it would >>> be >>> > easy if i were dealing with an arraylist because i could just say >>> > arraylist.size. How would i accomplish that with the writable >>> counterpart? >>> > >>> > On Sun, Oct 19, 2008 at 1:04 AM, Joman Chu <[EMAIL PROTECTED]> wrote: >>> > >>> >> Hi, >>> >> >>> >> For the ArrayList object, try taking a look at the implementation of >>> >> ArrayListWritable by Jimmy Lin at UMD here: >>> >> >>> >> >>> >> >>> https://subversion.umiacs.umd.edu/umd-hadoop/core/trunk/src/edu/umd/cloud9/io/ArrayListWritable.java >>> >> >>> >> But basically in the readFields methods, I prefer using each Writable >>> >> object's readFields method to read the data in. For example, for your >>> >> double variable, I would use a DoubleWritable object and in the >>> >> MyWritable.readFields(DataInput in), I would use >>> >> nameofdoublewritable.readFields(in). For the >>> >> MyWritable.write(DataOutput out) method, I would use >>> >> nameofdoublewritable.write(out). >>> >> >>> >> Have a good one, >>> >> >>> >> Joman Chu >>> >> http://www.notatypewriter.com/ >>> >> AIM: ARcanUSNUMquam >>> >> >>> >> >>> >> >>> >> On Sun, Oct 19, 2008 at 3:30 AM, Yih Sun Khoo <[EMAIL PROTECTED]> >>> wrote: >>> >> > I don't quite know how to write the read and write functions, but I >>> want >>> >> to >>> >> > write my own writable, which should have a DoubleWritable/double >>> value >>> >> > followed by a list of Strings/Text. This Writable will be used as a >>> >> value. >>> >> > Is the code below the best way to go about writing such a writable? >>> >> > >>> >> > import java.io.DataInput; >>> >> > import java.io.DataOutput; >>> >> > import java.io.EOFException; >>> >> > import java.io.IOException; >>> >> > import java.util.ArrayList; >>> >> > >>> >> > import org.apache.hadoop.io.Writable; >>> >> > >>> >> > public class MyWritable implements Writable { >>> >> >private double score; >>> >> >private ArrayList nameList; >>> >> > >>> >> >public void setScore(double score) { >>> >> >this.score= score; >>> >> >} >>> >> > >>> >> >public void setNameList(ArrayList nameList) { >>> >> >this.nameList= nameList; >>> >> >} >>> >> > >>> >> >public double getScore() { >>> >> >return score; >>> >> >} >>> >> > >>> >> >public ArrayList getNameList() { >>> >> >return nameList; >>> >> >} >>> >> > >>> >> >public void readFields(DataInput in) throws IOException { >>> >> >score= in.readDouble(); >>> >> >try { >>> >> >do { >>> >> >nameList.add(in.readUTF()); >>> >> >} while (true); >>> >> >} catch (EOFException eofe) { >>> >> >// continue; done >>> >> >} >>> >> >} >>> >> > >>> >> >public void write(DataOutput out) throws IOException { >>> >> >out.writeDouble(score); >>> >> >for (String name: nameList) { >>> >> >out.writ
Re: How do I implement a Writable into another Writable?
The Hadoop internals should handle that for you. Once the execution enters the reduce() method in your class that implements the Reducer interface, all of the data should have been read into the appropriate objects by the reducer. Hadoop itself will use the readFields() method you wrote in your own Writable to accomplish this. If you've ever used any of Hadoop's own Writables, such as DoubleWritable or IntWritable in a MapReduce job, it is exactly the same thing. Hadoop's internals will transparently read the data into the DoubleWritable or IntWritable object and present it to you in reduce(). Joman Chu http://www.notatypewriter.com/ AIM: ARcanUSNUMquam On Sun, Oct 19, 2008 at 4:32 AM, Yih Sun Khoo <[EMAIL PROTECTED]> wrote: > Let's say in the reduce phase your value happens to hold an > ArrayListWritable > In this example, value is of type ArrayListWritable > Maybe I've not thought about this or done this before, but how does one > "read data in from the DataInput stream" in the reduce phase so that the > ArrayListWritable which is a value already passed to the reducer can be used > as ArrayListWritable > > On Sun, Oct 19, 2008 at 1:25 AM, Joman Chu <[EMAIL PROTECTED]> wrote: > >> Since the ArrayListWritable extends ArrayList, you have access to all >> the ArrayList methods as well. Once you read data in from the >> DataInput stream, you should be able to use ArrayListWritable just >> like a regular ArrayList. >> Joman Chu >> http://www.notatypewriter.com/ >> AIM: ARcanUSNUMquam >> >> >> >> On Sun, Oct 19, 2008 at 4:16 AM, Yih Sun Khoo <[EMAIL PROTECTED]> wrote: >> > Hmm, what method from ArrayListWritable allows you to access the >> different >> > elements of the ArrayList? Would it be readFields? for example, in a >> > reduce phase, if I needed to know the size of the array list, it would be >> > easy if i were dealing with an arraylist because i could just say >> > arraylist.size. How would i accomplish that with the writable >> counterpart? >> > >> > On Sun, Oct 19, 2008 at 1:04 AM, Joman Chu <[EMAIL PROTECTED]> wrote: >> > >> >> Hi, >> >> >> >> For the ArrayList object, try taking a look at the implementation of >> >> ArrayListWritable by Jimmy Lin at UMD here: >> >> >> >> >> >> >> https://subversion.umiacs.umd.edu/umd-hadoop/core/trunk/src/edu/umd/cloud9/io/ArrayListWritable.java >> >> >> >> But basically in the readFields methods, I prefer using each Writable >> >> object's readFields method to read the data in. For example, for your >> >> double variable, I would use a DoubleWritable object and in the >> >> MyWritable.readFields(DataInput in), I would use >> >> nameofdoublewritable.readFields(in). For the >> >> MyWritable.write(DataOutput out) method, I would use >> >> nameofdoublewritable.write(out). >> >> >> >> Have a good one, >> >> >> >> Joman Chu >> >> http://www.notatypewriter.com/ >> >> AIM: ARcanUSNUMquam >> >> >> >> >> >> >> >> On Sun, Oct 19, 2008 at 3:30 AM, Yih Sun Khoo <[EMAIL PROTECTED]> wrote: >> >> > I don't quite know how to write the read and write functions, but I >> want >> >> to >> >> > write my own writable, which should have a DoubleWritable/double value >> >> > followed by a list of Strings/Text. This Writable will be used as a >> >> value. >> >> > Is the code below the best way to go about writing such a writable? >> >> > >> >> > import java.io.DataInput; >> >> > import java.io.DataOutput; >> >> > import java.io.EOFException; >> >> > import java.io.IOException; >> >> > import java.util.ArrayList; >> >> > >> >> > import org.apache.hadoop.io.Writable; >> >> > >> >> > public class MyWritable implements Writable { >> >> >private double score; >> >> >private ArrayList nameList; >> >> > >> >> >public void setScore(double score) { >> >> >this.score= score; >> >> >} >> >> > >> >> >public void setNameList(ArrayList nameList) { >> >> >this.nameList= nameList; >> >> >} >> >> > >> >> >public double getScore() { >> >> >return score; >> >> >} >> >> > >> >> >public ArrayList getNameList() { >> >> >return nameList; >> >> >} >> >> > >> >> >public void readFields(DataInput in) throws IOException { >> >> >score= in.readDouble(); >> >> >try { >> >> >do { >> >> >nameList.add(in.readUTF()); >> >> >} while (true); >> >> >} catch (EOFException eofe) { >> >> >// continue; done >> >> >} >> >> >} >> >> > >> >> >public void write(DataOutput out) throws IOException { >> >> >out.writeDouble(score); >> >> >for (String name: nameList) { >> >> >out.writeUTF(name); >> >> >} >> >> >} >> >> > } >> >> > >> >> >> > >> >
Re: How do I implement a Writable into another Writable?
Hey Joman, public void readFields(DataInput in) throws IOException { doubleWritable.readFields(in); tupleWritable.readFields(in); } It seems this line of code is giving me some problems doubleWritable.readFields(in); The error stack ... 08/10/19 01:55:06 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: bytes 08/10/19 01:55:06 WARN mapred.LocalJobRunner: job_local_0001 java.lang.NullPointerException at test.TestWritable.readFields(TestWritable.java:36) at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:67) at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:40) at org.apache.hadoop.mapred.Task$ValuesIterator.readNextValue(Task.java:738) .. On Sun, Oct 19, 2008 at 1:40 AM, Yih Sun Khoo <[EMAIL PROTECTED]> wrote: > Also, I've noticed TupleWritable to be quite useful. > What are good techniques for using TupleWritable in a mapping phase for a > "list of Text" when you do not know the size of that "list" ahead of time > > Say I had a custom writable which implemented TupleWritable and the custom > writable contained a setter method > mycustomwritable.setTupleWritable( ... ) > > Where the ellipsis is, there lies the TupleWritable. However I'm wondering > since TupleWritable can be constructed using TupleWritable(Writable[]), how > do I dynamically resize the Writable[] and add Text elements to it when I > don't know the size of the Writable[] very well. Does this make sense? > > > > On Sun, Oct 19, 2008 at 1:32 AM, Yih Sun Khoo <[EMAIL PROTECTED]> wrote: > >> Let's say in the reduce phase your value happens to hold an >> ArrayListWritable >> In this example, value is of type ArrayListWritable >> Maybe I've not thought about this or done this before, but how does one >> "read data in from the DataInput stream" in the reduce phase so that the >> ArrayListWritable which is a value already passed to the reducer can be used >> as ArrayListWritable >> >> >> On Sun, Oct 19, 2008 at 1:25 AM, Joman Chu <[EMAIL PROTECTED]> wrote: >> >>> Since the ArrayListWritable extends ArrayList, you have access to all >>> the ArrayList methods as well. Once you read data in from the >>> DataInput stream, you should be able to use ArrayListWritable just >>> like a regular ArrayList. >>> Joman Chu >>> http://www.notatypewriter.com/ >>> AIM: ARcanUSNUMquam >>> >>> >>> >>> On Sun, Oct 19, 2008 at 4:16 AM, Yih Sun Khoo <[EMAIL PROTECTED]> wrote: >>> > Hmm, what method from ArrayListWritable allows you to access the >>> different >>> > elements of the ArrayList? Would it be readFields? for example, in a >>> > reduce phase, if I needed to know the size of the array list, it would >>> be >>> > easy if i were dealing with an arraylist because i could just say >>> > arraylist.size. How would i accomplish that with the writable >>> counterpart? >>> > >>> > On Sun, Oct 19, 2008 at 1:04 AM, Joman Chu <[EMAIL PROTECTED]> wrote: >>> > >>> >> Hi, >>> >> >>> >> For the ArrayList object, try taking a look at the implementation of >>> >> ArrayListWritable by Jimmy Lin at UMD here: >>> >> >>> >> >>> >> >>> https://subversion.umiacs.umd.edu/umd-hadoop/core/trunk/src/edu/umd/cloud9/io/ArrayListWritable.java >>> >> >>> >> But basically in the readFields methods, I prefer using each Writable >>> >> object's readFields method to read the data in. For example, for your >>> >> double variable, I would use a DoubleWritable object and in the >>> >> MyWritable.readFields(DataInput in), I would use >>> >> nameofdoublewritable.readFields(in). For the >>> >> MyWritable.write(DataOutput out) method, I would use >>> >> nameofdoublewritable.write(out). >>> >> >>> >> Have a good one, >>> >> >>> >> Joman Chu >>> >> http://www.notatypewriter.com/ >>> >> AIM: ARcanUSNUMquam >>> >> >>> >> >>> >> >>> >> On Sun, Oct 19, 2008 at 3:30 AM, Yih Sun Khoo <[EMAIL PROTECTED]> >>> wrote: >>> >> > I don't quite know how to write the read and write functions, but I >>> want >>> >> to >>> >> > write my own writable, which should have a DoubleWritable/double >>> value >>> >> > followed by a list of Strings/Text. This Writable will be used as a >>> >> value. >>> >> > Is the code below the best way to go about writing such a writable? >>> >> > >>> >> > import java.io.DataInput; >>> >> > import java.io.DataOutput; >>> >> > import java.io.EOFException; >>> >> > import java.io.IOException; >>> >> > import java.util.ArrayList; >>> >> > >>> >> > import org.apache.hadoop.io.Writable; >>> >> > >>> >> > public class MyWritable implements Writable { >>> >> >private double score; >>> >> >private ArrayList nameList; >>> >> > >>> >> >public void setScore(double score) { >>> >> >this.score= score; >>> >> >} >>> >> > >>> >> >public void setNameList(ArrayList nameList) { >>> >> >this.nameList= nameList; >>> >> >} >>> >> > >>> >> >public double
Re: How do I implement a Writable into another Writable?
Also, I've noticed TupleWritable to be quite useful. What are good techniques for using TupleWritable in a mapping phase for a "list of Text" when you do not know the size of that "list" ahead of time Say I had a custom writable which implemented TupleWritable and the custom writable contained a setter method mycustomwritable.setTupleWritable( ... ) Where the ellipsis is, there lies the TupleWritable. However I'm wondering since TupleWritable can be constructed using TupleWritable(Writable[]), how do I dynamically resize the Writable[] and add Text elements to it when I don't know the size of the Writable[] very well. Does this make sense? On Sun, Oct 19, 2008 at 1:32 AM, Yih Sun Khoo <[EMAIL PROTECTED]> wrote: > Let's say in the reduce phase your value happens to hold an > ArrayListWritable > In this example, value is of type ArrayListWritable > Maybe I've not thought about this or done this before, but how does one > "read data in from the DataInput stream" in the reduce phase so that the > ArrayListWritable which is a value already passed to the reducer can be used > as ArrayListWritable > > > On Sun, Oct 19, 2008 at 1:25 AM, Joman Chu <[EMAIL PROTECTED]> wrote: > >> Since the ArrayListWritable extends ArrayList, you have access to all >> the ArrayList methods as well. Once you read data in from the >> DataInput stream, you should be able to use ArrayListWritable just >> like a regular ArrayList. >> Joman Chu >> http://www.notatypewriter.com/ >> AIM: ARcanUSNUMquam >> >> >> >> On Sun, Oct 19, 2008 at 4:16 AM, Yih Sun Khoo <[EMAIL PROTECTED]> wrote: >> > Hmm, what method from ArrayListWritable allows you to access the >> different >> > elements of the ArrayList? Would it be readFields? for example, in a >> > reduce phase, if I needed to know the size of the array list, it would >> be >> > easy if i were dealing with an arraylist because i could just say >> > arraylist.size. How would i accomplish that with the writable >> counterpart? >> > >> > On Sun, Oct 19, 2008 at 1:04 AM, Joman Chu <[EMAIL PROTECTED]> wrote: >> > >> >> Hi, >> >> >> >> For the ArrayList object, try taking a look at the implementation of >> >> ArrayListWritable by Jimmy Lin at UMD here: >> >> >> >> >> >> >> https://subversion.umiacs.umd.edu/umd-hadoop/core/trunk/src/edu/umd/cloud9/io/ArrayListWritable.java >> >> >> >> But basically in the readFields methods, I prefer using each Writable >> >> object's readFields method to read the data in. For example, for your >> >> double variable, I would use a DoubleWritable object and in the >> >> MyWritable.readFields(DataInput in), I would use >> >> nameofdoublewritable.readFields(in). For the >> >> MyWritable.write(DataOutput out) method, I would use >> >> nameofdoublewritable.write(out). >> >> >> >> Have a good one, >> >> >> >> Joman Chu >> >> http://www.notatypewriter.com/ >> >> AIM: ARcanUSNUMquam >> >> >> >> >> >> >> >> On Sun, Oct 19, 2008 at 3:30 AM, Yih Sun Khoo <[EMAIL PROTECTED]> >> wrote: >> >> > I don't quite know how to write the read and write functions, but I >> want >> >> to >> >> > write my own writable, which should have a DoubleWritable/double >> value >> >> > followed by a list of Strings/Text. This Writable will be used as a >> >> value. >> >> > Is the code below the best way to go about writing such a writable? >> >> > >> >> > import java.io.DataInput; >> >> > import java.io.DataOutput; >> >> > import java.io.EOFException; >> >> > import java.io.IOException; >> >> > import java.util.ArrayList; >> >> > >> >> > import org.apache.hadoop.io.Writable; >> >> > >> >> > public class MyWritable implements Writable { >> >> >private double score; >> >> >private ArrayList nameList; >> >> > >> >> >public void setScore(double score) { >> >> >this.score= score; >> >> >} >> >> > >> >> >public void setNameList(ArrayList nameList) { >> >> >this.nameList= nameList; >> >> >} >> >> > >> >> >public double getScore() { >> >> >return score; >> >> >} >> >> > >> >> >public ArrayList getNameList() { >> >> >return nameList; >> >> >} >> >> > >> >> >public void readFields(DataInput in) throws IOException { >> >> >score= in.readDouble(); >> >> >try { >> >> >do { >> >> >nameList.add(in.readUTF()); >> >> >} while (true); >> >> >} catch (EOFException eofe) { >> >> >// continue; done >> >> >} >> >> >} >> >> > >> >> >public void write(DataOutput out) throws IOException { >> >> >out.writeDouble(score); >> >> >for (String name: nameList) { >> >> >out.writeUTF(name); >> >> >} >> >> >} >> >> > } >> >> > >> >> >> > >> > >
Re: How do I implement a Writable into another Writable?
Let's say in the reduce phase your value happens to hold an ArrayListWritable In this example, value is of type ArrayListWritable Maybe I've not thought about this or done this before, but how does one "read data in from the DataInput stream" in the reduce phase so that the ArrayListWritable which is a value already passed to the reducer can be used as ArrayListWritable On Sun, Oct 19, 2008 at 1:25 AM, Joman Chu <[EMAIL PROTECTED]> wrote: > Since the ArrayListWritable extends ArrayList, you have access to all > the ArrayList methods as well. Once you read data in from the > DataInput stream, you should be able to use ArrayListWritable just > like a regular ArrayList. > Joman Chu > http://www.notatypewriter.com/ > AIM: ARcanUSNUMquam > > > > On Sun, Oct 19, 2008 at 4:16 AM, Yih Sun Khoo <[EMAIL PROTECTED]> wrote: > > Hmm, what method from ArrayListWritable allows you to access the > different > > elements of the ArrayList? Would it be readFields? for example, in a > > reduce phase, if I needed to know the size of the array list, it would be > > easy if i were dealing with an arraylist because i could just say > > arraylist.size. How would i accomplish that with the writable > counterpart? > > > > On Sun, Oct 19, 2008 at 1:04 AM, Joman Chu <[EMAIL PROTECTED]> wrote: > > > >> Hi, > >> > >> For the ArrayList object, try taking a look at the implementation of > >> ArrayListWritable by Jimmy Lin at UMD here: > >> > >> > >> > https://subversion.umiacs.umd.edu/umd-hadoop/core/trunk/src/edu/umd/cloud9/io/ArrayListWritable.java > >> > >> But basically in the readFields methods, I prefer using each Writable > >> object's readFields method to read the data in. For example, for your > >> double variable, I would use a DoubleWritable object and in the > >> MyWritable.readFields(DataInput in), I would use > >> nameofdoublewritable.readFields(in). For the > >> MyWritable.write(DataOutput out) method, I would use > >> nameofdoublewritable.write(out). > >> > >> Have a good one, > >> > >> Joman Chu > >> http://www.notatypewriter.com/ > >> AIM: ARcanUSNUMquam > >> > >> > >> > >> On Sun, Oct 19, 2008 at 3:30 AM, Yih Sun Khoo <[EMAIL PROTECTED]> wrote: > >> > I don't quite know how to write the read and write functions, but I > want > >> to > >> > write my own writable, which should have a DoubleWritable/double value > >> > followed by a list of Strings/Text. This Writable will be used as a > >> value. > >> > Is the code below the best way to go about writing such a writable? > >> > > >> > import java.io.DataInput; > >> > import java.io.DataOutput; > >> > import java.io.EOFException; > >> > import java.io.IOException; > >> > import java.util.ArrayList; > >> > > >> > import org.apache.hadoop.io.Writable; > >> > > >> > public class MyWritable implements Writable { > >> >private double score; > >> >private ArrayList nameList; > >> > > >> >public void setScore(double score) { > >> >this.score= score; > >> >} > >> > > >> >public void setNameList(ArrayList nameList) { > >> >this.nameList= nameList; > >> >} > >> > > >> >public double getScore() { > >> >return score; > >> >} > >> > > >> >public ArrayList getNameList() { > >> >return nameList; > >> >} > >> > > >> >public void readFields(DataInput in) throws IOException { > >> >score= in.readDouble(); > >> >try { > >> >do { > >> >nameList.add(in.readUTF()); > >> >} while (true); > >> >} catch (EOFException eofe) { > >> >// continue; done > >> >} > >> >} > >> > > >> >public void write(DataOutput out) throws IOException { > >> >out.writeDouble(score); > >> >for (String name: nameList) { > >> >out.writeUTF(name); > >> >} > >> >} > >> > } > >> > > >> > > >
Re: How do I implement a Writable into another Writable?
Since the ArrayListWritable extends ArrayList, you have access to all the ArrayList methods as well. Once you read data in from the DataInput stream, you should be able to use ArrayListWritable just like a regular ArrayList. Joman Chu http://www.notatypewriter.com/ AIM: ARcanUSNUMquam On Sun, Oct 19, 2008 at 4:16 AM, Yih Sun Khoo <[EMAIL PROTECTED]> wrote: > Hmm, what method from ArrayListWritable allows you to access the different > elements of the ArrayList? Would it be readFields? for example, in a > reduce phase, if I needed to know the size of the array list, it would be > easy if i were dealing with an arraylist because i could just say > arraylist.size. How would i accomplish that with the writable counterpart? > > On Sun, Oct 19, 2008 at 1:04 AM, Joman Chu <[EMAIL PROTECTED]> wrote: > >> Hi, >> >> For the ArrayList object, try taking a look at the implementation of >> ArrayListWritable by Jimmy Lin at UMD here: >> >> >> https://subversion.umiacs.umd.edu/umd-hadoop/core/trunk/src/edu/umd/cloud9/io/ArrayListWritable.java >> >> But basically in the readFields methods, I prefer using each Writable >> object's readFields method to read the data in. For example, for your >> double variable, I would use a DoubleWritable object and in the >> MyWritable.readFields(DataInput in), I would use >> nameofdoublewritable.readFields(in). For the >> MyWritable.write(DataOutput out) method, I would use >> nameofdoublewritable.write(out). >> >> Have a good one, >> >> Joman Chu >> http://www.notatypewriter.com/ >> AIM: ARcanUSNUMquam >> >> >> >> On Sun, Oct 19, 2008 at 3:30 AM, Yih Sun Khoo <[EMAIL PROTECTED]> wrote: >> > I don't quite know how to write the read and write functions, but I want >> to >> > write my own writable, which should have a DoubleWritable/double value >> > followed by a list of Strings/Text. This Writable will be used as a >> value. >> > Is the code below the best way to go about writing such a writable? >> > >> > import java.io.DataInput; >> > import java.io.DataOutput; >> > import java.io.EOFException; >> > import java.io.IOException; >> > import java.util.ArrayList; >> > >> > import org.apache.hadoop.io.Writable; >> > >> > public class MyWritable implements Writable { >> >private double score; >> >private ArrayList nameList; >> > >> >public void setScore(double score) { >> >this.score= score; >> >} >> > >> >public void setNameList(ArrayList nameList) { >> >this.nameList= nameList; >> >} >> > >> >public double getScore() { >> >return score; >> >} >> > >> >public ArrayList getNameList() { >> >return nameList; >> >} >> > >> >public void readFields(DataInput in) throws IOException { >> >score= in.readDouble(); >> >try { >> >do { >> >nameList.add(in.readUTF()); >> >} while (true); >> >} catch (EOFException eofe) { >> >// continue; done >> >} >> >} >> > >> >public void write(DataOutput out) throws IOException { >> >out.writeDouble(score); >> >for (String name: nameList) { >> >out.writeUTF(name); >> >} >> >} >> > } >> > >> >
Re: How do I implement a Writable into another Writable?
Hmm, what method from ArrayListWritable allows you to access the different elements of the ArrayList? Would it be readFields? for example, in a reduce phase, if I needed to know the size of the array list, it would be easy if i were dealing with an arraylist because i could just say arraylist.size. How would i accomplish that with the writable counterpart? On Sun, Oct 19, 2008 at 1:04 AM, Joman Chu <[EMAIL PROTECTED]> wrote: > Hi, > > For the ArrayList object, try taking a look at the implementation of > ArrayListWritable by Jimmy Lin at UMD here: > > > https://subversion.umiacs.umd.edu/umd-hadoop/core/trunk/src/edu/umd/cloud9/io/ArrayListWritable.java > > But basically in the readFields methods, I prefer using each Writable > object's readFields method to read the data in. For example, for your > double variable, I would use a DoubleWritable object and in the > MyWritable.readFields(DataInput in), I would use > nameofdoublewritable.readFields(in). For the > MyWritable.write(DataOutput out) method, I would use > nameofdoublewritable.write(out). > > Have a good one, > > Joman Chu > http://www.notatypewriter.com/ > AIM: ARcanUSNUMquam > > > > On Sun, Oct 19, 2008 at 3:30 AM, Yih Sun Khoo <[EMAIL PROTECTED]> wrote: > > I don't quite know how to write the read and write functions, but I want > to > > write my own writable, which should have a DoubleWritable/double value > > followed by a list of Strings/Text. This Writable will be used as a > value. > > Is the code below the best way to go about writing such a writable? > > > > import java.io.DataInput; > > import java.io.DataOutput; > > import java.io.EOFException; > > import java.io.IOException; > > import java.util.ArrayList; > > > > import org.apache.hadoop.io.Writable; > > > > public class MyWritable implements Writable { > >private double score; > >private ArrayList nameList; > > > >public void setScore(double score) { > >this.score= score; > >} > > > >public void setNameList(ArrayList nameList) { > >this.nameList= nameList; > >} > > > >public double getScore() { > >return score; > >} > > > >public ArrayList getNameList() { > >return nameList; > >} > > > >public void readFields(DataInput in) throws IOException { > >score= in.readDouble(); > >try { > >do { > >nameList.add(in.readUTF()); > >} while (true); > >} catch (EOFException eofe) { > >// continue; done > >} > >} > > > >public void write(DataOutput out) throws IOException { > >out.writeDouble(score); > >for (String name: nameList) { > >out.writeUTF(name); > >} > >} > > } > > >
Re: How do I implement a Writable into another Writable?
Hi, For the ArrayList object, try taking a look at the implementation of ArrayListWritable by Jimmy Lin at UMD here: https://subversion.umiacs.umd.edu/umd-hadoop/core/trunk/src/edu/umd/cloud9/io/ArrayListWritable.java But basically in the readFields methods, I prefer using each Writable object's readFields method to read the data in. For example, for your double variable, I would use a DoubleWritable object and in the MyWritable.readFields(DataInput in), I would use nameofdoublewritable.readFields(in). For the MyWritable.write(DataOutput out) method, I would use nameofdoublewritable.write(out). Have a good one, Joman Chu http://www.notatypewriter.com/ AIM: ARcanUSNUMquam On Sun, Oct 19, 2008 at 3:30 AM, Yih Sun Khoo <[EMAIL PROTECTED]> wrote: > I don't quite know how to write the read and write functions, but I want to > write my own writable, which should have a DoubleWritable/double value > followed by a list of Strings/Text. This Writable will be used as a value. > Is the code below the best way to go about writing such a writable? > > import java.io.DataInput; > import java.io.DataOutput; > import java.io.EOFException; > import java.io.IOException; > import java.util.ArrayList; > > import org.apache.hadoop.io.Writable; > > public class MyWritable implements Writable { >private double score; >private ArrayList nameList; > >public void setScore(double score) { >this.score= score; >} > >public void setNameList(ArrayList nameList) { >this.nameList= nameList; >} > >public double getScore() { >return score; >} > >public ArrayList getNameList() { >return nameList; >} > >public void readFields(DataInput in) throws IOException { >score= in.readDouble(); >try { >do { >nameList.add(in.readUTF()); >} while (true); >} catch (EOFException eofe) { >// continue; done >} >} > >public void write(DataOutput out) throws IOException { >out.writeDouble(score); >for (String name: nameList) { >out.writeUTF(name); >} >} > } >