[ https://issues.apache.org/jira/browse/SPARK-30260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
chenliang updated SPARK-30260: ------------------------------ Fix Version/s: 2.3.0 > Spark-Shell throw ClassNotFoundException exception for more than one > statement to use UDF jar > --------------------------------------------------------------------------------------------- > > Key: SPARK-30260 > URL: https://issues.apache.org/jira/browse/SPARK-30260 > Project: Spark > Issue Type: Bug > Components: Spark Shell, SQL > Affects Versions: 2.2.0, 2.3.0, 2.4.3, 2.4.4 > Reporter: chenliang > Priority: Major > Fix For: 2.3.0, 2.4.3 > > > When we start spark-shell and use the udf for the first statement ,it's ok. > But for the other statements it failed to load jar to current classpath and > would throw ClassNotFoundException,the problem can be reproduced as described > in the below. > {code:java} > scala> val res = spark.sql("select bigdata_test.Add(1,2)").show() > ---------------------- > |bigdata_test.Add(1, 2)| > ---------------------- > | 3| > ---------------------- > scala> val res = spark.sql("select bigdata_test.Add(1,2)").show() > org.apache.spark.sql.AnalysisException: No handler for UDF/UDAF/UDTF > 'scala.didi.udf.Add': java.lang.ClassNotFoundException: scala.didi.udf.Add; > line 1 pos 8 > at > scala.reflect.internal.util.AbstractFileClassLoader.findClass(AbstractFileClassLoader.scala:62) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at > org.apache.spark.sql.hive.HiveShim$HiveFunctionWrapper.createFunction(HiveShim.scala:251) > at > org.apache.spark.sql.hive.HiveSimpleUDF.function$lzycompute(hiveUDFs.scala:56) > at org.apache.spark.sql.hive.HiveSimpleUDF.function(hiveUDFs.scala:56) > at > org.apache.spark.sql.hive.HiveSimpleUDF.method$lzycompute(hiveUDFs.scala:60) > at org.apache.spark.sql.hive.HiveSimpleUDF.method(hiveUDFs.scala:59) > at > org.apache.spark.sql.hive.HiveSimpleUDF.dataType$lzycompute(hiveUDFs.scala:77) > at org.apache.spark.sql.hive.HiveSimpleUDF.dataType(hiveUDFs.scala:77) > at > org.apache.spark.sql.hive.HiveSessionCatalog$$anonfun$makeFunctionExpression$3.apply(HiveSessionCatalog.scala:79) > at > org.apache.spark.sql.hive.HiveSessionCatalog$$anonfun$makeFunctionExpression$3.apply(HiveSessionCatalog.scala:71) > at scala.util.Try.getOrElse(Try.scala:79) > at > org.apache.spark.sql.hive.HiveSessionCatalog.makeFunctionExpression(HiveSessionCatalog.scala:71) > at > org.apache.spark.sql.catalyst.catalog.SessionCatalog$$anonfun$org$apache$spark$sql$catalyst$catalog$SessionCatalog$$makeFunctionBuilder$1.apply(SessionCatalog.scala:1133){code} -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org