This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push: new 37d19b9ee0e4 [SPARK-46237][SQL][TESTS] Make `HiveDDLSuite` independently testable 37d19b9ee0e4 is described below commit 37d19b9ee0e4e100e37358e71d771a2e42d01d88 Author: yangjie01 <yangji...@baidu.com> AuthorDate: Mon Dec 4 08:52:23 2023 -0800 [SPARK-46237][SQL][TESTS] Make `HiveDDLSuite` independently testable ### What changes were proposed in this pull request? When I test `HiveDDLSuite` with ``` build/sbt "hive/testOnly org.apache.spark.sql.hive.execution.HiveDDLSuite" -Phive ``` This test throws an error: ``` [info] - SPARK-34261: Avoid side effect if create exists temporary function *** FAILED *** (4 milliseconds) [info] java.util.NoSuchElementException: key not found: default [info] at scala.collection.MapOps.default(Map.scala:274) [info] at scala.collection.MapOps.default$(Map.scala:273) [info] at scala.collection.AbstractMap.default(Map.scala:405) [info] at scala.collection.MapOps.apply(Map.scala:176) [info] at scala.collection.MapOps.apply$(Map.scala:175) [info] at scala.collection.AbstractMap.apply(Map.scala:405) [info] at org.apache.spark.sql.hive.execution.HiveDDLSuite.$anonfun$new$445(HiveDDLSuite.scala:3275) [info] at org.apache.spark.sql.test.SQLTestUtilsBase.withUserDefinedFunction(SQLTestUtils.scala:256) [info] at org.apache.spark.sql.test.SQLTestUtilsBase.withUserDefinedFunction$(SQLTestUtils.scala:254) [info] at org.apache.spark.sql.execution.command.DDLSuite.withUserDefinedFunction(DDLSuite.scala:326) [info] at org.apache.spark.sql.hive.execution.HiveDDLSuite.$anonfun$new$444(HiveDDLSuite.scala:3267) ``` I manually printed the content of `spark.sparkContext.addedJars`, which is an empty `Map`. However, when I execute ``` build/sbt "hive/testOnly org.apache.spark.sql.hive.execution.SQLQuerySuite org.apache.spark.sql.hive.execution.HiveDDLSuite" -Phive ``` All tests pass, and the content of `spark.sparkContext.addedJars` is ``` Map(default -> Map(spark://localhost:54875/jars/SPARK-21101-1.0.jar -> 1701676986594, spark://localhost:54875/jars/hive-contrib-2.3.9.jar -> 1701676944590, spark://localhost:54875/jars/TestUDTF.jar -> 1701676921340)) ``` The reason why this failure is not reproduced in the GitHub Action test is because `SQLQuerySuite` is indeed executed before `HiveDDLSuite`. So in the current PR, I change to use `.get("default").foreach(_.remove(k))` that the remove operation is only performed when `.get("default")` is not `None`. ### Why are the changes needed? Make `HiveDDLSuite` independently testable. ### Does this PR introduce _any_ user-facing change? No, just for test ### How was this patch tested? - Pass Github Actions - Manual check `HiveDDLSuite` with this pr and all test passed ### Was this patch authored or co-authored using generative AI tooling? No Closes #44153 from LuciferYang/HiveDDLSuite. Authored-by: yangjie01 <yangji...@baidu.com> Signed-off-by: Dongjoon Hyun <dh...@apple.com> --- .../test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala b/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala index c3a528da382a..2f5d1fcbb540 100644 --- a/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala +++ b/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala @@ -3270,7 +3270,7 @@ class HiveDDLSuite val jarName = "TestUDTF.jar" val jar = spark.asInstanceOf[TestHiveSparkSession].getHiveFile(jarName).toURI.toString spark.sparkContext.allAddedJars.keys.find(_.contains(jarName)) - .foreach(spark.sparkContext.addedJars("default").remove) + .foreach(k => spark.sparkContext.addedJars.get("default").foreach(_.remove(k))) assert(!spark.sparkContext.listJars().exists(_.contains(jarName))) val e = intercept[AnalysisException] { sql("CREATE TEMPORARY FUNCTION f1 AS " + --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org