Hi Josh, it doesn't have to. However, your table does not specify any input format, which makes Hive use the default HiveIgnoreKeyTextOutputFormat (see http://grepcode.com/file/repository.cloudera.com/content/repositories/releases/org.apache.hadoop.hive/hive-exec/0.7.1-cdh3u1/org/apache/hadoop/hive/ql/io/HiveIgnoreKeyTextOutputFormat.java ).
What's the bigger picture, meaning, what is the actual format you want to write down to disk ? If you're storing it in text format, your SerDe's serialize method has to return either a Text or a BytesWritable, as you can see in the code linked earlier. It looks like it's returning a DocumentWritable object, which HiveIgnoreKeyTextOutputFormat does not know how to handle. R. On Mon, Jul 29, 2013 at 5:42 AM, Clum, Josh <josh.c...@merck.com> wrote: > Hi, > > I wrote a custom serde that stores takes data and stores it in objects > called DocumentWritable that inherits from Writable. When I try to run code > to insert into the table that uses my serde, I get an error saying that > DocumentWritable cannot be cast to BytesWritable. Does my DocumentWritable > need to inherit from org.apache.hadoop.io.BytesWritable? > > *I'm creating a table using a custom serde that I wrote:* > > CREATE TABLE avro_test ( > NOTEBOOK_ID STRING, > … > DLC_XID STRING > ) > ROW FORMAT SERDE 'com.merck.huron.hive.LuceneSerde'; > > *When I try to insert data using a select statement I am storing it into > a class I have called DocumentWritable:* > > INSERT OVERWRITE TABLE avro_test SELECT * FROM avro > > *DocumentWritable inherits from Writable. When I do the select and my > serde code gets executed I get the following error:* > > Error: java.lang.RuntimeException: > org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while > processing row > {"notebook_id":"1","stuff_id":"1","person_id":"0","person_role_id":"1","reference":"NO > DOC","lastname":"OMITTED","firstinitial":"TBD","source_id":null,"source_name":"MRL > SYNTHESIS","notebook":null,"pg":null,"firstname":null,"middlename":null,"site":null,"mail":null,"dlc_xid":null} > at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:161) > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:158) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1441) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:153) > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime > Error while processing row > {"notebook_id":"1","stuff_id":"1","person_id":"0","person_role_id":"1","reference":"NO > DOC","lastname":"OMITTED","firstinitial":"TBD","source_id":null,"source_name":"MRL > SYNTHESIS","notebook":null,"pg":null,"firstname":null,"middlename":null,"site":null,"mail":null,"dlc_xid":null} > at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:565) > at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:143) > ... 8 more > Caused by: java.lang.ClassCastException: > com.comp.proj.indexing.*DocumentWritable > cannot be cast to org.apache.hadoop.io.BytesWritable* > at > org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat$1.write(HiveIgnoreKeyTextOutputFormat.java:90) > at > org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:606) > at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474) > at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800) > at > org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84) > at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474) > at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800) > at > org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:83) > at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474) > at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800) > at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:546) > ... 9 more > > Notice: This e-mail message, together with any attachments, contains > information of Merck & Co., Inc. (One Merck Drive, Whitehouse Station, > New Jersey, USA 08889), and/or its affiliates Direct contact information > for affiliates is available at > http://www.merck.com/contact/contacts.html) that may be confidential, > proprietary copyrighted and/or legally privileged. It is intended solely > for the use of the individual or entity named on this message. If you are > not the intended recipient, and have received this message in error, > please notify us immediately by reply e-mail and then delete it from > your system. > -- ---------------------------------------------------------- Good judgement comes with experience. Experience comes with bad judgement. ---------------------------------------------------------- Roberto Congiu - Data Engineer - OpenX tel: +1 626 466 1141