[
https://issues.apache.org/jira/browse/SPARK-56764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18083581#comment-18083581
]
Anupam Yadav commented on SPARK-56764:
--------------------------------------
Closing the PR after discussion with sarutak. The original scenario (Thrift
Server) works correctly via parent classloader delegation (SparkSubmit sets up
the classloader chain). The only gap is programmatic SparkSession creation
without SparkSubmit, which sarutak confirmed is not an intended use case. This
is not a legitimate bug.
> CANNOT_LOAD_FUNCTION_CLASS in a new session
> -------------------------------------------
>
> Key: SPARK-56764
> URL: https://issues.apache.org/jira/browse/SPARK-56764
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.1.1
> Reporter: pin_zhang
> Priority: Major
> Labels: pull-request-available
> Attachments: udf.jar
>
>
> 1. ./sbin/start-thriftserver.sh with async=false and create a UDF
> spark.sql.hive.thriftServer.async false
> spark.jars
> file:///home/test/spark-4.1.1-bin-hadoop3-connect/custom/udf.jar
> CREATE FUNCTION IF NOT EXISTS hello AS 'com.test.HelloUDF';
> 2. restart thriftserver
> ./bin/beeline -u jdbc:hive2://localhost:10000 -e 'select 1;'
> ./bin/beeline -u jdbc:hive2://localhost:10000 -e 'select hello("a");'
> Connecting to jdbc:hive2://localhost:10000
> Connected to: Spark SQL (version 4.1.1)
> Driver: Hive JDBC (version 2.3.10)
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> Error: Error running query: [CANNOT_LOAD_FUNCTION_CLASS]
> org.apache.spark.sql.AnalysisException: [CANNOT_LOAD_FUNCTION_CLASS]
> Cannot load class com.test.HelloUDF when registering the function
> `spark_catalog`.`default`.`hello`, please make sure it is on the classpath.
> SQLSTATE: 46103; line 1 pos 7 (state=46103,code=0)
> Closing: 0: jdbc:hive2://localhost:10000
> 3. why set async=false, because pyhive client collect to hive in sync mode
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
