Upgrade to CDH 5.5 for spark. It should work
On Sat, Jan 9, 2016 at 12:17 AM, Ophir Etzion <op...@foursquare.com> wrote:
> It didn't work. assuming I did the right thing.
> in the properties you could see
>
> {"key":"hive.aux.jars.path","value":"file:///data/loko/foursquare.web-hiverc/current/hadoop-hive-serde.jar,file:///data/loko/foursquare.web-hiverc/current/hadoop-hive-udf.jar","isFinal":false,"resource":"programatically"}
> which includes the jar that has the class I need but I still get
>
> org.apache.hive.com.esotericsoftware.kryo.KryoException: Unable to find
> class: com.foursquare.hadoop.hive.io.HiveThriftSequenceFileInputFormat
>
>
>
> On Fri, Jan 8, 2016 at 12:24 PM, Edward Capriolo <edlinuxg...@gmail.com>
> wrote:
>
>> You can not 'add jar' input formats and serde's. They need to be part of
>> your auxlib.
>>
>> On Fri, Jan 8, 2016 at 12:19 PM, Ophir Etzion <op...@foursquare.com>
>> wrote:
>>
>>> I tried now. still getting
>>>
>>> 16/01/08 16:37:34 ERROR exec.Utilities: Failed to load plan:
>>> hdfs://hadoop-alidoro-nn-vip/tmp/hive/hive/c2af9882-38a9-42b0-8d17-3f56708383e8/hive_2016-01-08_16-36-41_370_3307331506800215903-3/-mr-10004/3c90a796-47fc-4541-bbec-b196c40aefab/map.xml:
>>> org.apache.hive.com.esotericsoftware.kryo.KryoException: Unable to find
>>> class: com.foursquare.hadoop.hive.io.HiveThriftSequenceFileInputFormat
>>> Serialization trace:
>>> inputFileFormatClass (org.apache.hadoop.hive.ql.plan.PartitionDesc)
>>> aliasToPartnInfo (org.apache.hadoop.hive.ql.plan.MapWork)
>>> org.apache.hive.com.esotericsoftware.kryo.KryoException: Unable to find
>>> class: com.foursquare.hadoop.hive.io.HiveThriftSequenceFileInputFormat
>>>
>>>
>>> HiveThriftSequenceFileInputFormat is in one of the jars I'm trying to add.
>>>
>>>
>>> On Thu, Jan 7, 2016 at 9:58 PM, Prem Sure <premsure...@gmail.com> wrote:
>>>
>>>> did you try -- jars property in spark submit? if your jar is of huge
>>>> size, you can pre-load the jar on all executors in a common available
>>>> directory to avoid network IO.
>>>>
>>>> On Thu, Jan 7, 2016 at 4:03 PM, Ophir Etzion <op...@foursquare.com>
>>>> wrote:
>>>>
>>>>> I' trying to add jars before running a query using hive on spark on
>>>>> cdh 5.4.3.
>>>>> I've tried applying the patch in
>>>>> https://issues.apache.org/jira/browse/HIVE-12045 (manually as the
>>>>> patch is done on a different hive version) but still hasn't succeeded.
>>>>>
>>>>> did anyone manage to do ADD JAR successfully with CDH?
>>>>>
>>>>> Thanks,
>>>>> Ophir
>>>>>
>>>>
>>>>
>>>
>>
>
