ArrayWritable usage
I'm having some difficulty with using ArrayWritable in the following test code: ArrayWritable array = new ArrayWritable(IntWritable.class); IntWritable[] ints = new IntWritable[4]; for (int i =0 ; i < 4; i++) { ints[i] = new IntWritable(i); } array.set(ints); writer.append(new LongWritable(1), array); writer.close(); After starting the map reduce application's test, I'm getting the following errors which I suspect are due to improper use of ArrayWritable. Can someone tell me if the code given above is correct? java.lang.RuntimeException: java.lang.NoSuchMethodException: org.apache.hadoop.io.ArrayWritable.() at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:115) at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:62) at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:40) at org.apache.hadoop.io.SequenceFile$Reader.deserializeValue(SequenceFile.java:1817) at org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.java:1790) at org.apache.hadoop.mapreduce.lib.input.SequenceFileRecordReader.nextKeyValue(SequenceFileRecordReader.java:74) at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:531) at org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369) at org.apache.hadoop.mapred.Child$4.run(Child.java:259) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059) at org.apache.hadoop.mapred.Child.main(Child.java:253) Caused by: java.lang.NoSuchMethodException: org.apache.hadoop.io.ArrayWritable.() at java.lang.Class.getConstructor0(Class.java:2706) at java.lang.Class.getDeclaredConstructor(Class.java:1985) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:109) ... 15 more
Re: ArrayWritable usage
Hi Dhruv, Is this working well for you ?? Are you able to do IntWritable [] abc = array.get(); I am trying similar thing for IntTwoDArrayWritable. The array.set works but array.get returns Writable[][] and I am not able to cast it to IntWritable[][]. -- View this message in context: http://lucene.472066.n3.nabble.com/ArrayWritable-usage-tp3138520p3576386.html Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
Re: ArrayWritable usage
Hi, ArrayWritable is a touch hard to use. Say you have an array of IntWritable[]. The get() method or ArrayWritable, after serializations/deserialization, does in fact return an array of type Writable. As such you cannot cast it directly to IntWritable[]. Individual elements are of type IntWritable and can be cast as such. Will not work: IntWritable[] array = (IntWritable[]) writable.get(); Will work: for(Writable element : writable.get()) { IntWritable intWritable = (IntWritable)element; } Brock On Sat, Dec 10, 2011 at 3:58 PM, zanurag wrote: > Hi Dhruv, > Is this working well for you ?? Are you able to do IntWritable [] abc = > array.get(); > > I am trying similar thing for IntTwoDArrayWritable. > The array.set works but array.get returns Writable[][] and I am not able > to cast it to IntWritable[][]. > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/ArrayWritable-usage-tp3138520p3576386.html > Sent from the Hadoop lucene-users mailing list archive at Nabble.com. >
Re: ArrayWritable usage
ArrayWritable doesn't serialize type information. You need to subclass it (e.g. IntArrayWritable) and create a no arg constructor which calls super(IntWritable.class). Use this instead of ArrayWritable directly. If you want to store more than one type, look at the source for MapWritable to see how it generates type codes. -Joey On Jul 4, 2011 2:55 PM, "Dhruv Kumar" wrote: > I'm having some difficulty with using ArrayWritable in the following test > code: > > ArrayWritable array = new ArrayWritable(IntWritable.class); > > IntWritable[] ints = new IntWritable[4]; > > for (int i =0 ; i < 4; i++) { > ints[i] = new IntWritable(i); > > } > > array.set(ints); > > writer.append(new LongWritable(1), array); > > writer.close(); > > After starting the map reduce application's test, I'm getting the following > errors which I suspect are due to improper use of ArrayWritable. Can someone > tell me if the code given above is correct? > > java.lang.RuntimeException: java.lang.NoSuchMethodException: > org.apache.hadoop.io.ArrayWritable.() > at > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:115) > at > org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:62) > at > org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:40) > at > org.apache.hadoop.io.SequenceFile$Reader.deserializeValue(SequenceFile.java:1817) > at > org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.java:1790) > at > org.apache.hadoop.mapreduce.lib.input.SequenceFileRecordReader.nextKeyValue(SequenceFileRecordReader.java:74) > at > org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:531) > at > org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369) > at org.apache.hadoop.mapred.Child$4.run(Child.java:259) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059) > at org.apache.hadoop.mapred.Child.main(Child.java:253) > Caused by: java.lang.NoSuchMethodException: > org.apache.hadoop.io.ArrayWritable.() > at java.lang.Class.getConstructor0(Class.java:2706) > at java.lang.Class.getDeclaredConstructor(Class.java:1985) > at > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:109) > ... 15 more
Re: ArrayWritable usage
Thanks, your answer and the documentation at http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/io/ArrayWritable.htmlfixed the issue. For others benefit, I have reproduced the solution below. Modified test code: SequenceFile.Writer writer = SequenceFile.createWriter(fs, conf, new Path("/home/input/in"), LongWritable.class, IntArrayWritable.class); IntArrayWritable array = new IntArrayWritable(); IntWritable[] ints = new IntWritable[4]; for (int i =0 ; i < 4; i++) { ints[i] = new IntWritable(i); } array.set(ints); writer.append(new LongWritable(1), array); writer.close(); New IntArrayWritableClass: public class IntArrayWritable extends ArrayWritable { public IntArrayWritable() { super(IntWritable.class); } } On Mon, Jul 4, 2011 at 3:00 PM, Joey Echeverria wrote: > ArrayWritable doesn't serialize type information. You need to subclass it > (e.g. IntArrayWritable) and create a no arg constructor which calls > super(IntWritable.class). > > Use this instead of ArrayWritable directly. If you want to store more than > one type, look at the source for MapWritable to see how it generates type > codes. > > -Joey > On Jul 4, 2011 2:55 PM, "Dhruv Kumar" wrote: > > I'm having some difficulty with using ArrayWritable in the following test > > code: > > > > ArrayWritable array = new ArrayWritable(IntWritable.class); > > > > IntWritable[] ints = new IntWritable[4]; > > > > for (int i =0 ; i < 4; i++) { > > ints[i] = new IntWritable(i); > > > > } > > > > array.set(ints); > > > > writer.append(new LongWritable(1), array); > > > > writer.close(); > > > > After starting the map reduce application's test, I'm getting the > following > > errors which I suspect are due to improper use of ArrayWritable. Can > someone > > tell me if the code given above is correct? > > > > java.lang.RuntimeException: java.lang.NoSuchMethodException: > > org.apache.hadoop.io.ArrayWritable.() > > at > > > > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:115) > > at > > > > org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:62) > > at > > > > org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:40) > > at > > > > org.apache.hadoop.io.SequenceFile$Reader.deserializeValue(SequenceFile.java:1817) > > at > > > > org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.java:1790) > > at > > > > org.apache.hadoop.mapreduce.lib.input.SequenceFileRecordReader.nextKeyValue(SequenceFileRecordReader.java:74) > > at > > > > org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:531) > > at > > org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67) > > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143) > > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763) > > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369) > > at org.apache.hadoop.mapred.Child$4.run(Child.java:259) > > at java.security.AccessController.doPrivileged(Native Method) > > at javax.security.auth.Subject.doAs(Subject.java:396) > > at > > > > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059) > > at org.apache.hadoop.mapred.Child.main(Child.java:253) > > Caused by: java.lang.NoSuchMethodException: > > org.apache.hadoop.io.ArrayWritable.() > > at java.lang.Class.getConstructor0(Class.java:2706) > > at java.lang.Class.getDeclaredConstructor(Class.java:1985) > > at > > > > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:109) > > ... 15 more >