[
https://issues.apache.org/jira/browse/SPARK-26454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16731818#comment-16731818
]
Hyukjin Kwon commented on SPARK-26454:
--------------------------------------
Which Spark version do you use? Looks line number is different from Spark 2.3.2:
https://github.com/apache/spark/blob/v2.3.2/core/src/main/scala/org/apache/spark/SparkContext.scala#L1810-L1858
{code}
java.lang.IllegalArgumentException: requirement failed: File custom.jar was
already registered with a different path (old path =
/opt/sparkclient/Spark2x/tmp/spark-10ea8f59-fa23-46c5-af12-aa029bf2f5cb/custom.jar,
new path =
/opt/sparkclient/Spark2x/tmp/spark-ed12eb5e-b7b9-49d0-a7a4-a0dba9141ac9/custom.jar
at scala.Predef$.require(Predef.scala:224)
at
org.apache.spark.rpc.netty.NettyStreamManager.addJar(NettyStreamManager.scala:78)
at org.apache.spark.SparkContext.addJarFile$1(SparkContext.scala:1829)
at org.apache.spark.SparkContext.addJar(SparkContext.scala:1851)
at
org.apache.spark.sql.internal.SessionResourceLoader.addJar(SessionState.scala:189)
at
org.apache.spark.sql.hive.HiveSessionResourceLoader.addJar(HiveSessionStateBuilder.scala:119)
at
org.apache.spark.sql.hive.HiveACLSessionResourceLoader.addJar(HiveACLSessionStateBuilder.scala:110)
at
org.apache.spark.sql.internal.SessionResourceLoader.loadResource(SessionState.scala:157)
at
org.apache.spark.sql.catalyst.catalog.SessionCatalog$$anonfun$loadFunctionResources$1.apply(SessionCatalo
{code}
Assuming from the codes, it should just show the error log and the code should
work.
> IllegegalArgument Exception is Thrown while creating new UDF with JAR
> ---------------------------------------------------------------------
>
> Key: SPARK-26454
> URL: https://issues.apache.org/jira/browse/SPARK-26454
> Project: Spark
> Issue Type: Bug
> Components: Spark Shell
> Affects Versions: 2.3.2
> Reporter: Udbhav Agrawal
> Priority: Major
> Attachments: create_exception.txt
>
>
> 【Test step】：
> 1.launch spark-shell
> 2. set role admin;
> 3. create new function
> CREATE FUNCTION Func AS
> 'org.apache.hadoop.hive.ql.udf.generic.GenericUDFLastDayTest' USING JAR
> 'hdfs:///tmp/super_udf/two_udfs.jar'
> 4. Do select on the function
> sql("select Func('2018-03-09')").show()
> 5.Create new UDF with same JAR
> sql("CREATE FUNCTION newFunc AS
> 'org.apache.hadoop.hive.ql.udf.generic.GenericUDFLastDayTest' USING JAR
> 'hdfs:///tmp/super_udf/two_udfs.jar'")
> 6. Do select on the new function created.
> sql("select newFunc ('2018-03-09')").show()
> 【Output】:
> Function is getting created but illegal argument exception is thrown , select
> provides result but with illegal argument exception.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
