Attila Zsolt Piros created SPARK-52988:
------------------------------------------
Summary: Race condition at the metastore function handling in
SessionCatalog
Key: SPARK-52988
URL: https://issues.apache.org/jira/browse/SPARK-52988
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.0.0, 3.5.6, 4.1.0
Reporter: Attila Zsolt Piros
Assignee: Attila Zsolt Piros
Even when "CREATE FUNCTION IF NOT EXISTS" is used in parallel a run can fail
with the following exception:
{noformat}
2025-07-25 01:22:21,731 [AA-Rule-ThreadPoolExec-2] ERROR ***** - An error
occured :
org.apache.spark.sql.AnalysisException: Function default.SparkTestUDF already
exists; line 1 pos 6734
at
org.apache.spark.sql.errors.QueryCompilationErrors$.functionAlreadyExistsError(QueryCompilationErrors.scala:654)
at
org.apache.spark.sql.catalyst.catalog.SessionCatalog.registerFunction(SessionCatalog.scala:1487)
at
org.apache.spark.sql.catalyst.catalog.SessionCatalog.resolvePersistentFunctionInternal(SessionCatalog.scala:1719)
at
org.apache.spark.sql.catalyst.catalog.SessionCatalog.resolvePersistentFunction(SessionCatalog.scala:1675)
at
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveFunctions$.org$apache$spark$sql$catalyst$analysis$Analyzer$ResolveFunctions$$resolveV1Function(Analyzer.scala:2284)
at
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveFunctions$$anonfun$apply$25$$anonfun$applyOrElse$103.$anonfun$applyOrElse$109(Analyzer.scala:2234)
at scala.Option.getOrElse(Option.scala:189)
at
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveFunctions$$anonfun$apply$25$$anonfun$applyOrElse$103.$anonfun$applyOrElse$108(Analyzer.scala:2231)
at
org.apache.spark.sql.catalyst.analysis.package$.withPosition(package.scala:60)
at
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveFunctions$$anonfun$apply$25$$anonfun$applyOrElse$103.applyOrElse(Analyzer.scala:2231)
at
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveFunctions$$anonfun$apply$25$$anonfun$applyOrElse$103.applyOrElse(Analyzer.scala:2205)
...
{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
