Hi Josh,
it doesn't have to. However, your table does not specify any input format,
which makes Hive use the default HiveIgnoreKeyTextOutputFormat (see
http://grepcode.com/file/repository.cloudera.com/content/repositories/releases/org.apache.hadoop.hive/hive-exec/0.7.1-cdh3u1/org/apache/hadoop/hive/ql/io/HiveIgnoreKeyTextOutputFormat.java
).
What's the bigger picture, meaning, what is the actual format you want to
write down to disk ?
If you're storing it in text format, your SerDe's serialize method has to
return either a Text or a BytesWritable, as you can see in the code linked
earlier. It looks like it's returning a DocumentWritable object, which
HiveIgnoreKeyTextOutputFormat does not know how to handle.
R.
On Mon, Jul 29, 2013 at 5:42 AM, Clum, Josh <josh.c...@merck.com> wrote:
> Hi,
>
> I wrote a custom serde that stores takes data and stores it in objects
> called DocumentWritable that inherits from Writable. When I try to run code
> to insert into the table that uses my serde, I get an error saying that
> DocumentWritable cannot be cast to BytesWritable. Does my DocumentWritable
> need to inherit from org.apache.hadoop.io.BytesWritable?
>
> *I'm creating a table using a custom serde that I wrote:*
>
> CREATE TABLE avro_test (
> NOTEBOOK_ID STRING,
> …
> DLC_XID STRING
> )
> ROW FORMAT SERDE 'com.merck.huron.hive.LuceneSerde';
>
> *When I try to insert data using a select statement I am storing it into
> a class I have called DocumentWritable:*
>
> INSERT OVERWRITE TABLE avro_test SELECT * FROM avro
>
> *DocumentWritable inherits from Writable. When I do the select and my
> serde code gets executed I get the following error:*
>
> Error: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row
> {"notebook_id":"1","stuff_id":"1","person_id":"0","person_role_id":"1","reference":"NO
> DOC","lastname":"OMITTED","firstinitial":"TBD","source_id":null,"source_name":"MRL
> SYNTHESIS","notebook":null,"pg":null,"firstname":null,"middlename":null,"site":null,"mail":null,"dlc_xid":null}
> at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:161)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:158)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1441)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:153)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row
> {"notebook_id":"1","stuff_id":"1","person_id":"0","person_role_id":"1","reference":"NO
> DOC","lastname":"OMITTED","firstinitial":"TBD","source_id":null,"source_name":"MRL
> SYNTHESIS","notebook":null,"pg":null,"firstname":null,"middlename":null,"site":null,"mail":null,"dlc_xid":null}
> at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:565)
> at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:143)
> ... 8 more
> Caused by: java.lang.ClassCastException:
> com.comp.proj.indexing.*DocumentWritable
> cannot be cast to org.apache.hadoop.io.BytesWritable*
> at
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat$1.write(HiveIgnoreKeyTextOutputFormat.java:90)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:606)
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:83)
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
> at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:546)
> ... 9 more
>
> Notice: This e-mail message, together with any attachments, contains
> information of Merck & Co., Inc. (One Merck Drive, Whitehouse Station,
> New Jersey, USA 08889), and/or its affiliates Direct contact information
> for affiliates is available at
> http://www.merck.com/contact/contacts.html) that may be confidential,
> proprietary copyrighted and/or legally privileged. It is intended solely
> for the use of the individual or entity named on this message. If you are
> not the intended recipient, and have received this message in error,
> please notify us immediately by reply e-mail and then delete it from
> your system.
>
--
----------------------------------------------------------
Good judgement comes with experience.
Experience comes with bad judgement.
----------------------------------------------------------
Roberto Congiu - Data Engineer - OpenX
tel: +1 626 466 1141
