btw, this issue happens only with classes needed for the inputFormat. if
the input format is org.apache.hadoop.mapred.TextInputFormat and the serde
is from an additional jar it works just fine.
I don't want to upgrade cdh for this. also, if it should work on cdh5.5 why
is that. what patch fixes that? (cdh 5.5 is the same hive version as
cdh5.4. is it spark related and not hive?)
On Sun, Jan 10, 2016 at 9:26 AM, sandeep vura <sandeepv...@gmail.com> wrote:
> Upgrade to CDH 5.5 for spark. It should work
>
> On Sat, Jan 9, 2016 at 12:17 AM, Ophir Etzion <op...@foursquare.com>
> wrote:
>
>> It didn't work. assuming I did the right thing.
>> in the properties you could see
>>
>> {"key":"hive.aux.jars.path","value":"file:///data/loko/foursquare.web-hiverc/current/hadoop-hive-serde.jar,file:///data/loko/foursquare.web-hiverc/current/hadoop-hive-udf.jar","isFinal":false,"resource":"programatically"}
>> which includes the jar that has the class I need but I still get
>>
>> org.apache.hive.com.esotericsoftware.kryo.KryoException: Unable to find
>> class: com.foursquare.hadoop.hive.io.HiveThriftSequenceFileInputFormat
>>
>>
>>
>> On Fri, Jan 8, 2016 at 12:24 PM, Edward Capriolo <edlinuxg...@gmail.com>
>> wrote:
>>
>>> You can not 'add jar' input formats and serde's. They need to be part of
>>> your auxlib.
>>>
>>> On Fri, Jan 8, 2016 at 12:19 PM, Ophir Etzion <op...@foursquare.com>
>>> wrote:
>>>
>>>> I tried now. still getting
>>>>
>>>> 16/01/08 16:37:34 ERROR exec.Utilities: Failed to load plan:
>>>> hdfs://hadoop-alidoro-nn-vip/tmp/hive/hive/c2af9882-38a9-42b0-8d17-3f56708383e8/hive_2016-01-08_16-36-41_370_3307331506800215903-3/-mr-10004/3c90a796-47fc-4541-bbec-b196c40aefab/map.xml:
>>>> org.apache.hive.com.esotericsoftware.kryo.KryoException: Unable to find
>>>> class: com.foursquare.hadoop.hive.io.HiveThriftSequenceFileInputFormat
>>>> Serialization trace:
>>>> inputFileFormatClass (org.apache.hadoop.hive.ql.plan.PartitionDesc)
>>>> aliasToPartnInfo (org.apache.hadoop.hive.ql.plan.MapWork)
>>>> org.apache.hive.com.esotericsoftware.kryo.KryoException: Unable to find
>>>> class: com.foursquare.hadoop.hive.io.HiveThriftSequenceFileInputFormat
>>>>
>>>>
>>>> HiveThriftSequenceFileInputFormat is in one of the jars I'm trying to add.
>>>>
>>>>
>>>> On Thu, Jan 7, 2016 at 9:58 PM, Prem Sure <premsure...@gmail.com>
>>>> wrote:
>>>>
>>>>> did you try -- jars property in spark submit? if your jar is of huge
>>>>> size, you can pre-load the jar on all executors in a common available
>>>>> directory to avoid network IO.
>>>>>
>>>>> On Thu, Jan 7, 2016 at 4:03 PM, Ophir Etzion <op...@foursquare.com>
>>>>> wrote:
>>>>>
>>>>>> I' trying to add jars before running a query using hive on spark on
>>>>>> cdh 5.4.3.
>>>>>> I've tried applying the patch in
>>>>>> https://issues.apache.org/jira/browse/HIVE-12045 (manually as the
>>>>>> patch is done on a different hive version) but still hasn't succeeded.
>>>>>>
>>>>>> did anyone manage to do ADD JAR successfully with CDH?
>>>>>>
>>>>>> Thanks,
>>>>>> Ophir
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
