ArrayWritable usage

2011-07-04 Thread Dhruv Kumar 
I'm having some difficulty with using ArrayWritable in the following test
code:

ArrayWritable array = new ArrayWritable(IntWritable.class);

  IntWritable[] ints = new IntWritable[4];

  for (int i =0 ; i < 4; i++) {
ints[i] = new IntWritable(i);

  }

  array.set(ints);

  writer.append(new LongWritable(1), array);

  writer.close();

After starting the map reduce application's test, I'm getting the following
errors which I suspect are due to improper use of ArrayWritable. Can someone
tell me if the code given above is correct?

java.lang.RuntimeException: java.lang.NoSuchMethodException:
org.apache.hadoop.io.ArrayWritable.()
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:115)
at
org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:62)
at
org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:40)
at
org.apache.hadoop.io.SequenceFile$Reader.deserializeValue(SequenceFile.java:1817)
at
org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.java:1790)
at
org.apache.hadoop.mapreduce.lib.input.SequenceFileRecordReader.nextKeyValue(SequenceFileRecordReader.java:74)
at
org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:531)
at
org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
at org.apache.hadoop.mapred.Child.main(Child.java:253)
Caused by: java.lang.NoSuchMethodException:
org.apache.hadoop.io.ArrayWritable.()
at java.lang.Class.getConstructor0(Class.java:2706)
at java.lang.Class.getDeclaredConstructor(Class.java:1985)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:109)
... 15 more

Re: ArrayWritable usage

2011-12-12 Thread zanurag 
Hi Dhruv,
Is this working well for you ?? Are you able to do IntWritable [] abc =
array.get();

I am trying similar thing for IntTwoDArrayWritable.
The array.set works but array.get returns Writable[][] and I am not able
to cast it to IntWritable[][].

--
View this message in context: 
http://lucene.472066.n3.nabble.com/ArrayWritable-usage-tp3138520p3576386.html
Sent from the Hadoop lucene-users mailing list archive at Nabble.com.

Re: ArrayWritable usage

2011-12-13 Thread Brock Noland 
Hi,

ArrayWritable is a touch hard to use. Say you have an array of
IntWritable[]. The get() method or ArrayWritable, after
serializations/deserialization, does in fact return an array of type
Writable. As such you cannot cast it directly to IntWritable[]. Individual
elements are of type IntWritable and can be cast as such.

Will not work:

IntWritable[] array = (IntWritable[]) writable.get();

Will work:

for(Writable element : writable.get()) {
  IntWritable intWritable = (IntWritable)element;
}

Brock

On Sat, Dec 10, 2011 at 3:58 PM, zanurag  wrote:

> Hi Dhruv,
> Is this working well for you ?? Are you able to do IntWritable [] abc =
> array.get();
>
> I am trying similar thing for IntTwoDArrayWritable.
> The array.set works but array.get returns Writable[][] and I am not able
> to cast it to IntWritable[][].
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/ArrayWritable-usage-tp3138520p3576386.html
> Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
>

Re: ArrayWritable usage

2011-07-04 Thread Joey Echeverria 
ArrayWritable doesn't serialize type information. You need to subclass it
(e.g. IntArrayWritable) and create a no arg constructor which calls
super(IntWritable.class).

Use this instead of ArrayWritable directly. If you want to store more than
one type, look at the source for MapWritable to see how it generates type
codes.

-Joey
On Jul 4, 2011 2:55 PM, "Dhruv Kumar"  wrote:
> I'm having some difficulty with using ArrayWritable in the following test
> code:
>
> ArrayWritable array = new ArrayWritable(IntWritable.class);
>
> IntWritable[] ints = new IntWritable[4];
>
> for (int i =0 ; i < 4; i++) {
> ints[i] = new IntWritable(i);
>
> }
>
> array.set(ints);
>
> writer.append(new LongWritable(1), array);
>
> writer.close();
>
> After starting the map reduce application's test, I'm getting the
following
> errors which I suspect are due to improper use of ArrayWritable. Can
someone
> tell me if the code given above is correct?
>
> java.lang.RuntimeException: java.lang.NoSuchMethodException:
> org.apache.hadoop.io.ArrayWritable.()
> at
>
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:115)
> at
>
org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:62)
> at
>
org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:40)
> at
>
org.apache.hadoop.io.SequenceFile$Reader.deserializeValue(SequenceFile.java:1817)
> at
>
org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.java:1790)
> at
>
org.apache.hadoop.mapreduce.lib.input.SequenceFileRecordReader.nextKeyValue(SequenceFileRecordReader.java:74)
> at
>
org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:531)
> at
> org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
>
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
> at org.apache.hadoop.mapred.Child.main(Child.java:253)
> Caused by: java.lang.NoSuchMethodException:
> org.apache.hadoop.io.ArrayWritable.()
> at java.lang.Class.getConstructor0(Class.java:2706)
> at java.lang.Class.getDeclaredConstructor(Class.java:1985)
> at
>
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:109)
> ... 15 more

Re: ArrayWritable usage

2011-07-04 Thread Dhruv Kumar 
Thanks,  your answer and the documentation at
http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/io/ArrayWritable.htmlfixed
the issue.

For others benefit, I have reproduced the solution below.

Modified test code:

   SequenceFile.Writer writer = SequenceFile.createWriter(fs, conf, new
Path("/home/input/in"), LongWritable.class, IntArrayWritable.class);

  IntArrayWritable array = new IntArrayWritable();

  IntWritable[] ints = new IntWritable[4];

  for (int i =0 ; i < 4; i++) {
ints[i] = new IntWritable(i);

  }

  array.set(ints);

  writer.append(new LongWritable(1), array);

  writer.close();



New IntArrayWritableClass:

public class IntArrayWritable extends ArrayWritable {
  public IntArrayWritable() {
super(IntWritable.class);
  }
}


On Mon, Jul 4, 2011 at 3:00 PM, Joey Echeverria  wrote:

> ArrayWritable doesn't serialize type information. You need to subclass it
> (e.g. IntArrayWritable) and create a no arg constructor which calls
> super(IntWritable.class).
>
> Use this instead of ArrayWritable directly. If you want to store more than
> one type, look at the source for MapWritable to see how it generates type
> codes.
>
> -Joey
> On Jul 4, 2011 2:55 PM, "Dhruv Kumar"  wrote:
> > I'm having some difficulty with using ArrayWritable in the following test
> > code:
> >
> > ArrayWritable array = new ArrayWritable(IntWritable.class);
> >
> > IntWritable[] ints = new IntWritable[4];
> >
> > for (int i =0 ; i < 4; i++) {
> > ints[i] = new IntWritable(i);
> >
> > }
> >
> > array.set(ints);
> >
> > writer.append(new LongWritable(1), array);
> >
> > writer.close();
> >
> > After starting the map reduce application's test, I'm getting the
> following
> > errors which I suspect are due to improper use of ArrayWritable. Can
> someone
> > tell me if the code given above is correct?
> >
> > java.lang.RuntimeException: java.lang.NoSuchMethodException:
> > org.apache.hadoop.io.ArrayWritable.()
> > at
> >
>
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:115)
> > at
> >
>
> org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:62)
> > at
> >
>
> org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:40)
> > at
> >
>
> org.apache.hadoop.io.SequenceFile$Reader.deserializeValue(SequenceFile.java:1817)
> > at
> >
>
> org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.java:1790)
> > at
> >
>
> org.apache.hadoop.mapreduce.lib.input.SequenceFileRecordReader.nextKeyValue(SequenceFileRecordReader.java:74)
> > at
> >
>
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:531)
> > at
> > org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
> > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
> > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
> > at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:396)
> > at
> >
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
> > at org.apache.hadoop.mapred.Child.main(Child.java:253)
> > Caused by: java.lang.NoSuchMethodException:
> > org.apache.hadoop.io.ArrayWritable.()
> > at java.lang.Class.getConstructor0(Class.java:2706)
> > at java.lang.Class.getDeclaredConstructor(Class.java:1985)
> > at
> >
>
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:109)
> > ... 15 more
>