[
https://issues.apache.org/jira/browse/HIVE-9410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14305668#comment-14305668
]
Marcelo Vanzin commented on HIVE-9410:
--------------------------------------
Ah, I see, didn't notice that. Thanks for clarifying!
Depending on the exact semantics expected by Hive, I'd make one suggestion
though: instead of having a growing hierarchy of class loader every time you
call addJar() and run a job (then call addJar() then run another job then...),
I'd have a simple sub-class of URLClassLoader that exposes the protected
"addURL()" method. That way, you don't need to keep this jar list, and you
don't need to create this chain of class loaders - you just modify the existing
context class loader by adding the new URL.
> ClassNotFoundException occurs during hive query case execution with UDF
> defined [Spark Branch]
> ----------------------------------------------------------------------------------------------
>
> Key: HIVE-9410
> URL: https://issues.apache.org/jira/browse/HIVE-9410
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
> Environment: CentOS 6.5
> JDK1.7
> Reporter: Xin Hao
> Assignee: Chengxiang Li
> Fix For: spark-branch, 1.1.0
>
> Attachments: HIVE-9410.1-spark.patch, HIVE-9410.2-spark.patch,
> HIVE-9410.3-spark.patch, HIVE-9410.4-spark.patch, HIVE-9410.4-spark.patch
>
>
> We have a hive query case with UDF defined (i.e. BigBench case Q10, Q18
> etc.). It will be passed for default Hive (on MR) mode, while failed for Hive
> On Spark mode (both Standalone and Yarn-Client).
> Although we use 'add jar XXXX.jar;' to add the UDF jar explicitly, the issue
> still exists.
> BTW, if we put the UDF jar into $HIVE_HOME/lib dir, the case will be passed.
> Detail Error Message is as below (NOTE:
> de.bankmark.bigbench.queries.q10.SentimentUDF is the UDF which contained in
> jar bigbenchqueriesmr.jar, and we have add command like 'add jar
> /location/to/bigbenchqueriesmr.jar;' into .sql explicitly)
> {code}
> INFO [pool-1-thread-1]: client.RemoteDriver (RemoteDriver.java:call(316)) -
> Failed to run job 8dd120cb-1a4d-4d1c-ba31-61eac648c27d
> org.apache.hive.com.esotericsoftware.kryo.KryoException: Unable to find
> class: de.bankmark.bigbench.queries.q10.SentimentUDF
> Serialization trace:
> genericUDTF (org.apache.hadoop.hive.ql.plan.UDTFDesc)
> conf (org.apache.hadoop.hive.ql.exec.UDTFOperator)
> childOperators (org.apache.hadoop.hive.ql.exec.SelectOperator)
> childOperators (org.apache.hadoop.hive.ql.exec.MapJoinOperator)
> childOperators (org.apache.hadoop.hive.ql.exec.FilterOperator)
> childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
> aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
> right (org.apache.commons.lang3.tuple.ImmutablePair)
> edgeProperties (org.apache.hadoop.hive.ql.plan.SparkWork)
> at
> org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:138)
> at
> org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:115)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:656)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:99)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:112)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:112)
> ...
> Caused by: java.lang.ClassNotFoundException:
> de.bankmark.bigbench.queries.q10.SentimentUDF
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:270)
> at
> org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:136)
> ... 55 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
