Re: Hive On Spark - Using custom SerDe
It is better to use safety valve for hive configuration. On Mon, Nov 16, 2015 at 8:03 AM, Daniel Haviv < daniel.ha...@veracity-group.com> wrote: > Hi, > How should I set it ? just a normal set in hive or add it via the safety > valve to the hive or sparks configuartion? > > Thank you. > Daniel > > On Mon, Nov 16, 2015 at 5:46 PM, Jimmy Xiang wrote: > >> Have you add your class to "spark.kryo.classesToRegister"? You also need >> to make sure your jar is in ""hive.aux.jars.path". >> >> Thanks, >> Jimmy >> >> On Mon, Nov 16, 2015 at 1:44 AM, Daniel Haviv < >> daniel.ha...@veracity-group.com> wrote: >> >>> Hi, >>> We have a custom SerDe we would like to use with Hive on Spark but I'm >>> not sure how to. >>> The error messages are pretty clear about the fact that it can't find my >>> SerDE's class: >>> >>> Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException: Unable >>> to find class: com.mycompany.hive.WhaleAvroGenericInputFormat >>> >>> >>> >>> >>> Thank you. >>> >>> Daniel >>> >>> >> >
Re: Hive On Spark - Using custom SerDe
Hi, How should I set it ? just a normal set in hive or add it via the safety valve to the hive or sparks configuartion? Thank you. Daniel On Mon, Nov 16, 2015 at 5:46 PM, Jimmy Xiang wrote: > Have you add your class to "spark.kryo.classesToRegister"? You also need > to make sure your jar is in ""hive.aux.jars.path". > > Thanks, > Jimmy > > On Mon, Nov 16, 2015 at 1:44 AM, Daniel Haviv < > daniel.ha...@veracity-group.com> wrote: > >> Hi, >> We have a custom SerDe we would like to use with Hive on Spark but I'm >> not sure how to. >> The error messages are pretty clear about the fact that it can't find my >> SerDE's class: >> >> Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException: Unable >> to find class: com.mycompany.hive.WhaleAvroGenericInputFormat >> >> >> >> >> Thank you. >> >> Daniel >> >> >
Re: Hive On Spark - Using custom SerDe
Have you add your class to "spark.kryo.classesToRegister"? You also need to make sure your jar is in ""hive.aux.jars.path". Thanks, Jimmy On Mon, Nov 16, 2015 at 1:44 AM, Daniel Haviv < daniel.ha...@veracity-group.com> wrote: > Hi, > We have a custom SerDe we would like to use with Hive on Spark but I'm not > sure how to. > The error messages are pretty clear about the fact that it can't find my > SerDE's class: > > Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException: Unable to > find class: com.mycompany.hive.WhaleAvroGenericInputFormat > > > > > Thank you. > > Daniel > >
Hive On Spark - Using custom SerDe
Hi, We have a custom SerDe we would like to use with Hive on Spark but I'm not sure how to. The error messages are pretty clear about the fact that it can't find my SerDE's class: Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException: Unable to find class: com.mycompany.hive.WhaleAvroGenericInputFormat Thank you. Daniel
