[
https://issues.apache.org/jira/browse/SPARK-26624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen resolved SPARK-26624.
-------------------------------
Resolution: Duplicate
> Different classloader use on subsequent call to same query, causing different
> behavior
> --------------------------------------------------------------------------------------
>
> Key: SPARK-26624
> URL: https://issues.apache.org/jira/browse/SPARK-26624
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.2.0
> Reporter: Bimalendu Choudhary
> Priority: Major
>
> When calling a Hive UDF function from spark shell, we get the output when we
> call the query first time, but when we call the query again it gives
> following error
> #spark2-shell
> scala> spark.sql("select test(name) from customers limit 2").show (50, false)
> org.apache.spark.sql.AnalysisException: No handler for Hive UDF
> 'com.vnb.fgp.generic.udf.encrypt.EncryptGenericUDF':
> We have not provided the UDF jar files on the command line, but still we get
> the output. The function test is created in Hive service as a permanent
> function using the jar file.
> Debugging it further we see that on first invocation of the select command
> the following classLoader is being used and it has a path pointing to the
> hdfs directory as set in Hive service:
> loader:
> org.apache.spark.sql.internal.NonClosableMutableURLClassLoader@42cef0af
> hdfs:/tmp/bimal/hive-extensions-1.0-SNAPSHOT-jar-with-dependencies.jar
> file:/usr/java/jdk1.8.0_162/jre/lib/resources.jar
> file:/usr/java/jdk1.8.0_162/jre/lib/rt.jar
> On subsequent calls, a different class loader is being used:
> loader scala.tools.nsc.interpreter.IMain$TranslatingClassLoader@7bc3ec95
> file:/usr/java/jdk1.8.0_162/jre/lib/resources.jar
> file:/usr/java/jdk1.8.0_162/jre/lib/rt.jar
> file:/usr/java/jdk1.8.0_162/jre/lib/jsse.jar
> file:/usr/java/jdk1.8.0_162/jre/lib/jce.jar
> This does not have the hdfs path for the jar file and hence the exception is
> generated.
> Most probably the classloader is picking things from Hive metastore.
> If we pass the UDF jar files on command line using --jars option, everything
> works fine.
> But this indicates that the classLoader and classpaths are different when
> called first and second time causing inconsistent behavior and cause problem.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
