[
https://issues.apache.org/jira/browse/SPARK-37159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kousuke Saruta resolved SPARK-37159.
------------------------------------
Fix Version/s: 3.3.0
Resolution: Fixed
Issue resolved in https://github.com/apache/spark/pull/34425
> Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
> -----------------------------------------------------------------------
>
> Key: SPARK-37159
> URL: https://issues.apache.org/jira/browse/SPARK-37159
> Project: Spark
> Issue Type: Bug
> Components: SQL, Tests
> Affects Versions: 3.3.0
> Reporter: Kousuke Saruta
> Assignee: Kousuke Saruta
> Priority: Minor
> Fix For: 3.3.0
>
>
> SPARK-37105 seems to have fixed most of tests in `sql/hive` for Java 17 but
> `HiveExternalCatalogVersionsSuite`.
> {code}
> [info] org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite *** ABORTED
> *** (42 seconds, 526 milliseconds)
> [info] spark-submit returned with exit code 1.
> [info] Command line:
> '/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/test-spark-d86af275-0c40-4b47-9cab-defa92a5ffa7/spark-3.2.0/bin/spark-submit'
> '--name' 'prepare testing tables' '--master' 'local[2]' '--conf'
> 'spark.ui.enabled=false' '--conf' 'spark.master.rest.enabled=false' '--conf'
> 'spark.sql.hive.metastore.version=2.3' '--conf'
> 'spark.sql.hive.metastore.jars=maven' '--conf'
> 'spark.sql.warehouse.dir=/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/warehouse-69d9bdbc-54ce-443b-8677-a413663ddb62'
> '--conf' 'spark.sql.test.version.index=0' '--driver-java-options'
> '-Dderby.system.home=/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/warehouse-69d9bdbc-54ce-443b-8677-a413663ddb62'
>
> '/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/test15166225869206697603.py'
> [info]
> [info] 2021-10-28 06:07:18.486 - stderr> Using Spark's default log4j
> profile: org/apache/spark/log4j-defaults.properties
> [info] 2021-10-28 06:07:18.49 - stderr> 21/10/28 22:07:18 INFO
> SparkContext: Running Spark version 3.2.0
> [info] 2021-10-28 06:07:18.537 - stderr> 21/10/28 22:07:18 WARN
> NativeCodeLoader: Unable to load native-hadoop library for your platform...
> using builtin-java classes where applicable
> [info] 2021-10-28 06:07:18.616 - stderr> 21/10/28 22:07:18 INFO
> ResourceUtils: ==============================================================
> [info] 2021-10-28 06:07:18.616 - stderr> 21/10/28 22:07:18 INFO
> ResourceUtils: No custom resources configured for spark.driver.
> [info] 2021-10-28 06:07:18.616 - stderr> 21/10/28 22:07:18 INFO
> ResourceUtils: ==============================================================
> [info] 2021-10-28 06:07:18.617 - stderr> 21/10/28 22:07:18 INFO
> SparkContext: Submitted application: prepare testing tables
> [info] 2021-10-28 06:07:18.632 - stderr> 21/10/28 22:07:18 INFO
> ResourceProfile: Default ResourceProfile created, executor resources:
> Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name:
> memory, amount: 1024, script: , vendor: , offHeap -> name: offHeap, amount:
> 0, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0)
> [info] 2021-10-28 06:07:18.641 - stderr> 21/10/28 22:07:18 INFO
> ResourceProfile: Limiting resource is cpu
> [info] 2021-10-28 06:07:18.641 - stderr> 21/10/28 22:07:18 INFO
> ResourceProfileManager: Added ResourceProfile id: 0
> [info] 2021-10-28 06:07:18.679 - stderr> 21/10/28 22:07:18 INFO
> SecurityManager: Changing view acls to: kou
> [info] 2021-10-28 06:07:18.679 - stderr> 21/10/28 22:07:18 INFO
> SecurityManager: Changing modify acls to: kou
> [info] 2021-10-28 06:07:18.68 - stderr> 21/10/28 22:07:18 INFO
> SecurityManager: Changing view acls groups to:
> [info] 2021-10-28 06:07:18.68 - stderr> 21/10/28 22:07:18 INFO
> SecurityManager: Changing modify acls groups to:
> [info] 2021-10-28 06:07:18.68 - stderr> 21/10/28 22:07:18 INFO
> SecurityManager: SecurityManager: authentication disabled; ui acls disabled;
> users with view permissions: Set(kou); groups with view permissions: Set();
> users with modify permissions: Set(kou); groups with modify permissions:
> Set()
> [info] 2021-10-28 06:07:18.886 - stderr> 21/10/28 22:07:18 INFO Utils:
> Successfully started service 'sparkDriver' on port 35867.
> [info] 2021-10-28 06:07:18.906 - stderr> 21/10/28 22:07:18 INFO SparkEnv:
> Registering MapOutputTracker
> [info] 2021-10-28 06:07:18.93 - stderr> 21/10/28 22:07:18 INFO SparkEnv:
> Registering BlockManagerMaster
> [info] 2021-10-28 06:07:18.943 - stderr> 21/10/28 22:07:18 INFO
> BlockManagerMasterEndpoint: Using
> org.apache.spark.storage.DefaultTopologyMapper for getting topology
> information
> [info] 2021-10-28 06:07:18.944 - stderr> 21/10/28 22:07:18 INFO
> BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
> [info] 2021-10-28 06:07:18.945 - stdout> Traceback (most recent call last):
> [info] 2021-10-28 06:07:18.946 - stdout> File
> "/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/test15166225869206697603.py",
> line 5, in <module>
> [info] 2021-10-28 06:07:18.946 - stdout> spark =
> SparkSession.builder.enableHiveSupport().getOrCreate()
> [info] 2021-10-28 06:07:18.946 - stdout> File
> "/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/test-spark-d86af275-0c40-4b47-9cab-defa92a5ffa7/spark-3.2.0/python/lib/pyspark.zip/pyspark/sql/session.py",
> line 228, in getOrCreate
> [info] 2021-10-28 06:07:18.946 - stdout> File
> "/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/test-spark-d86af275-0c40-4b47-9cab-defa92a5ffa7/spark-3.2.0/python/lib/pyspark.zip/pyspark/context.py",
> line 392, in getOrCreate
> [info] 2021-10-28 06:07:18.946 - stdout> File
> "/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/test-spark-d86af275-0c40-4b47-9cab-defa92a5ffa7/spark-3.2.0/python/lib/pyspark.zip/pyspark/context.py",
> line 146, in __init__
> [info] 2021-10-28 06:07:18.946 - stdout> File
> "/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/test-spark-d86af275-0c40-4b47-9cab-defa92a5ffa7/spark-3.2.0/python/lib/pyspark.zip/pyspark/context.py",
> line 209, in _do_init
> [info] 2021-10-28 06:07:18.946 - stdout> File
> "/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/test-spark-d86af275-0c40-4b47-9cab-defa92a5ffa7/spark-3.2.0/python/lib/pyspark.zip/pyspark/context.py",
> line 329, in _initialize_context
> [info] 2021-10-28 06:07:18.946 - stdout> File
> "/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/test-spark-d86af275-0c40-4b47-9cab-defa92a5ffa7/spark-3.2.0/python/lib/py4j-0.10.9.2-src.zip/py4j/java_gateway.py",
> line 1573, in __call__
> [info] 2021-10-28 06:07:18.946 - stdout> File
> "/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/test-spark-d86af275-0c40-4b47-9cab-defa92a5ffa7/spark-3.2.0/python/lib/py4j-0.10.9.2-src.zip/py4j/protocol.py",
> line 326, in get_return_value
> [info] 2021-10-28 06:07:18.947 - stdout> py4j.protocol.Py4JJavaError: An
> error occurred while calling None.org.apache.spark.api.java.JavaSparkContext.
> [info] 2021-10-28 06:07:18.948 - stdout> : java.lang.IllegalAccessError:
> class org.apache.spark.storage.StorageUtils$ (in unnamed module @0x724b93a8)
> cannot access class sun.nio.ch.DirectBuffer (in module java.base) because
> module java.base does not export sun.nio.ch to unnamed module @0x724b93a8
> [info] 2021-10-28 06:07:18.948 - stdout> at
> org.apache.spark.storage.StorageUtils$.<init>(StorageUtils.scala:213)
> [info] 2021-10-28 06:07:18.948 - stdout> at
> org.apache.spark.storage.StorageUtils$.<clinit>(StorageUtils.scala)
> [info] 2021-10-28 06:07:18.948 - stdout> at
> org.apache.spark.storage.BlockManagerMasterEndpoint.<init>(BlockManagerMasterEndpoint.scala:110)
> [info] 2021-10-28 06:07:18.948 - stdout> at
> org.apache.spark.SparkEnv$.$anonfun$create$9(SparkEnv.scala:348)
> [info] 2021-10-28 06:07:18.948 - stdout> at
> org.apache.spark.SparkEnv$.registerOrLookupEndpoint$1(SparkEnv.scala:287)
> [info] 2021-10-28 06:07:18.948 - stdout> at
> org.apache.spark.SparkEnv$.create(SparkEnv.scala:336)
> [info] 2021-10-28 06:07:18.948 - stdout> at
> org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:191)
> [info] 2021-10-28 06:07:18.948 - stdout> at
> org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:277)
> [info] 2021-10-28 06:07:18.948 - stdout> at
> org.apache.spark.SparkContext.<init>(SparkContext.scala:460)
> [info] 2021-10-28 06:07:18.948 - stdout> at
> org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
> [info] 2021-10-28 06:07:18.948 - stdout> at
> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> [info] 2021-10-28 06:07:18.948 - stdout> at
> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
> [info] 2021-10-28 06:07:18.948 - stdout> at
> java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> [info] 2021-10-28 06:07:18.948 - stdout> at
> java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
> [info] 2021-10-28 06:07:18.948 - stdout> at
> java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
> [info] 2021-10-28 06:07:18.948 - stdout> at
> py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
> [info] 2021-10-28 06:07:18.948 - stdout> at
> py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
> [info] 2021-10-28 06:07:18.948 - stdout> at
> py4j.Gateway.invoke(Gateway.java:238)
> [info] 2021-10-28 06:07:18.948 - stdout> at
> py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
> [info] 2021-10-28 06:07:18.948 - stdout> at
> py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
> [info] 2021-10-28 06:07:18.948 - stdout> at
> py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
> [info] 2021-10-28 06:07:18.948 - stdout> at
> py4j.ClientServerConnection.run(ClientServerConnection.java:106)
> [info] 2021-10-28 06:07:18.948 - stdout> at
> java.base/java.lang.Thread.run(Thread.java:833)
> [info] 2021-10-28 06:07:18.948 - stdout>
> [info] 2021-10-28 06:07:18.975 - stderr> 21/10/28 22:07:18 INFO
> ShutdownHookManager: Shutdown hook called
> [info] 2021-10-28 06:07:18.976 - stderr> 21/10/28 22:07:18 INFO
> ShutdownHookManager: Deleting directory
> /tmp/spark-5bd38557-2b71-43f1-80ee-3648a456b3fc
> (SparkSubmitTestUtils.scala:96)
> [info] org.scalatest.exceptions.TestFailedException:
> [info] at
> org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)
> [info] at
> org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471)
> [info] at
> org.scalatest.funsuite.AnyFunSuite.newAssertionFailedException(AnyFunSuite.scala:1563)
> [info] at org.scalatest.Assertions.fail(Assertions.scala:933)
> [info] at org.scalatest.Assertions.fail$(Assertions.scala:929)
> [info] at org.scalatest.funsuite.AnyFunSuite.fail(AnyFunSuite.scala:1563)
> [info] at
> org.apache.spark.deploy.SparkSubmitTestUtils.runSparkSubmit(SparkSubmitTestUtils.scala:96)
> [info] at
> org.apache.spark.deploy.SparkSubmitTestUtils.runSparkSubmit$(SparkSubmitTestUtils.scala:43)
> [info] at
> org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite.runSparkSubmit(HiveExternalCatalogVersionsSuite.scala:56)
> [info] at
> org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite.$anonfun$beforeAll$2(HiveExternalCatalogVersionsSuite.scala:223)
> [info] at
> org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite.$anonfun$beforeAll$2$adapted(HiveExternalCatalogVersionsSuite.scala:204)
> [info] at
> scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
> [info] at
> scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
> [info] at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
> [info] at
> org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite.beforeAll(HiveExternalCatalogVersionsSuite.scala:204)
> [info] at
> org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
> [info] at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
> [info] at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
> [info] at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:62)
> [info] at
> org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:318)
> [info] at
> org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:513)
> [info] at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:413)
> [info] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> [info] at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> [info] at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> [info] at java.base/java.lang.Thread.run(Thread.java:833)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
