[ https://issues.apache.org/jira/browse/SPARK-37159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kousuke Saruta resolved SPARK-37159. ------------------------------------ Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved in https://github.com/apache/spark/pull/34425 > Change HiveExternalCatalogVersionsSuite to be able to test with Java 17 > ----------------------------------------------------------------------- > > Key: SPARK-37159 > URL: https://issues.apache.org/jira/browse/SPARK-37159 > Project: Spark > Issue Type: Bug > Components: SQL, Tests > Affects Versions: 3.3.0 > Reporter: Kousuke Saruta > Assignee: Kousuke Saruta > Priority: Minor > Fix For: 3.3.0 > > > SPARK-37105 seems to have fixed most of tests in `sql/hive` for Java 17 but > `HiveExternalCatalogVersionsSuite`. > {code} > [info] org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite *** ABORTED > *** (42 seconds, 526 milliseconds) > [info] spark-submit returned with exit code 1. > [info] Command line: > '/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/test-spark-d86af275-0c40-4b47-9cab-defa92a5ffa7/spark-3.2.0/bin/spark-submit' > '--name' 'prepare testing tables' '--master' 'local[2]' '--conf' > 'spark.ui.enabled=false' '--conf' 'spark.master.rest.enabled=false' '--conf' > 'spark.sql.hive.metastore.version=2.3' '--conf' > 'spark.sql.hive.metastore.jars=maven' '--conf' > 'spark.sql.warehouse.dir=/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/warehouse-69d9bdbc-54ce-443b-8677-a413663ddb62' > '--conf' 'spark.sql.test.version.index=0' '--driver-java-options' > '-Dderby.system.home=/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/warehouse-69d9bdbc-54ce-443b-8677-a413663ddb62' > > '/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/test15166225869206697603.py' > [info] > [info] 2021-10-28 06:07:18.486 - stderr> Using Spark's default log4j > profile: org/apache/spark/log4j-defaults.properties > [info] 2021-10-28 06:07:18.49 - stderr> 21/10/28 22:07:18 INFO > SparkContext: Running Spark version 3.2.0 > [info] 2021-10-28 06:07:18.537 - stderr> 21/10/28 22:07:18 WARN > NativeCodeLoader: Unable to load native-hadoop library for your platform... > using builtin-java classes where applicable > [info] 2021-10-28 06:07:18.616 - stderr> 21/10/28 22:07:18 INFO > ResourceUtils: ============================================================== > [info] 2021-10-28 06:07:18.616 - stderr> 21/10/28 22:07:18 INFO > ResourceUtils: No custom resources configured for spark.driver. > [info] 2021-10-28 06:07:18.616 - stderr> 21/10/28 22:07:18 INFO > ResourceUtils: ============================================================== > [info] 2021-10-28 06:07:18.617 - stderr> 21/10/28 22:07:18 INFO > SparkContext: Submitted application: prepare testing tables > [info] 2021-10-28 06:07:18.632 - stderr> 21/10/28 22:07:18 INFO > ResourceProfile: Default ResourceProfile created, executor resources: > Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: > memory, amount: 1024, script: , vendor: , offHeap -> name: offHeap, amount: > 0, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0) > [info] 2021-10-28 06:07:18.641 - stderr> 21/10/28 22:07:18 INFO > ResourceProfile: Limiting resource is cpu > [info] 2021-10-28 06:07:18.641 - stderr> 21/10/28 22:07:18 INFO > ResourceProfileManager: Added ResourceProfile id: 0 > [info] 2021-10-28 06:07:18.679 - stderr> 21/10/28 22:07:18 INFO > SecurityManager: Changing view acls to: kou > [info] 2021-10-28 06:07:18.679 - stderr> 21/10/28 22:07:18 INFO > SecurityManager: Changing modify acls to: kou > [info] 2021-10-28 06:07:18.68 - stderr> 21/10/28 22:07:18 INFO > SecurityManager: Changing view acls groups to: > [info] 2021-10-28 06:07:18.68 - stderr> 21/10/28 22:07:18 INFO > SecurityManager: Changing modify acls groups to: > [info] 2021-10-28 06:07:18.68 - stderr> 21/10/28 22:07:18 INFO > SecurityManager: SecurityManager: authentication disabled; ui acls disabled; > users with view permissions: Set(kou); groups with view permissions: Set(); > users with modify permissions: Set(kou); groups with modify permissions: > Set() > [info] 2021-10-28 06:07:18.886 - stderr> 21/10/28 22:07:18 INFO Utils: > Successfully started service 'sparkDriver' on port 35867. > [info] 2021-10-28 06:07:18.906 - stderr> 21/10/28 22:07:18 INFO SparkEnv: > Registering MapOutputTracker > [info] 2021-10-28 06:07:18.93 - stderr> 21/10/28 22:07:18 INFO SparkEnv: > Registering BlockManagerMaster > [info] 2021-10-28 06:07:18.943 - stderr> 21/10/28 22:07:18 INFO > BlockManagerMasterEndpoint: Using > org.apache.spark.storage.DefaultTopologyMapper for getting topology > information > [info] 2021-10-28 06:07:18.944 - stderr> 21/10/28 22:07:18 INFO > BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up > [info] 2021-10-28 06:07:18.945 - stdout> Traceback (most recent call last): > [info] 2021-10-28 06:07:18.946 - stdout> File > "/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/test15166225869206697603.py", > line 5, in <module> > [info] 2021-10-28 06:07:18.946 - stdout> spark = > SparkSession.builder.enableHiveSupport().getOrCreate() > [info] 2021-10-28 06:07:18.946 - stdout> File > "/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/test-spark-d86af275-0c40-4b47-9cab-defa92a5ffa7/spark-3.2.0/python/lib/pyspark.zip/pyspark/sql/session.py", > line 228, in getOrCreate > [info] 2021-10-28 06:07:18.946 - stdout> File > "/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/test-spark-d86af275-0c40-4b47-9cab-defa92a5ffa7/spark-3.2.0/python/lib/pyspark.zip/pyspark/context.py", > line 392, in getOrCreate > [info] 2021-10-28 06:07:18.946 - stdout> File > "/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/test-spark-d86af275-0c40-4b47-9cab-defa92a5ffa7/spark-3.2.0/python/lib/pyspark.zip/pyspark/context.py", > line 146, in __init__ > [info] 2021-10-28 06:07:18.946 - stdout> File > "/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/test-spark-d86af275-0c40-4b47-9cab-defa92a5ffa7/spark-3.2.0/python/lib/pyspark.zip/pyspark/context.py", > line 209, in _do_init > [info] 2021-10-28 06:07:18.946 - stdout> File > "/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/test-spark-d86af275-0c40-4b47-9cab-defa92a5ffa7/spark-3.2.0/python/lib/pyspark.zip/pyspark/context.py", > line 329, in _initialize_context > [info] 2021-10-28 06:07:18.946 - stdout> File > "/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/test-spark-d86af275-0c40-4b47-9cab-defa92a5ffa7/spark-3.2.0/python/lib/py4j-0.10.9.2-src.zip/py4j/java_gateway.py", > line 1573, in __call__ > [info] 2021-10-28 06:07:18.946 - stdout> File > "/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/test-spark-d86af275-0c40-4b47-9cab-defa92a5ffa7/spark-3.2.0/python/lib/py4j-0.10.9.2-src.zip/py4j/protocol.py", > line 326, in get_return_value > [info] 2021-10-28 06:07:18.947 - stdout> py4j.protocol.Py4JJavaError: An > error occurred while calling None.org.apache.spark.api.java.JavaSparkContext. > [info] 2021-10-28 06:07:18.948 - stdout> : java.lang.IllegalAccessError: > class org.apache.spark.storage.StorageUtils$ (in unnamed module @0x724b93a8) > cannot access class sun.nio.ch.DirectBuffer (in module java.base) because > module java.base does not export sun.nio.ch to unnamed module @0x724b93a8 > [info] 2021-10-28 06:07:18.948 - stdout> at > org.apache.spark.storage.StorageUtils$.<init>(StorageUtils.scala:213) > [info] 2021-10-28 06:07:18.948 - stdout> at > org.apache.spark.storage.StorageUtils$.<clinit>(StorageUtils.scala) > [info] 2021-10-28 06:07:18.948 - stdout> at > org.apache.spark.storage.BlockManagerMasterEndpoint.<init>(BlockManagerMasterEndpoint.scala:110) > [info] 2021-10-28 06:07:18.948 - stdout> at > org.apache.spark.SparkEnv$.$anonfun$create$9(SparkEnv.scala:348) > [info] 2021-10-28 06:07:18.948 - stdout> at > org.apache.spark.SparkEnv$.registerOrLookupEndpoint$1(SparkEnv.scala:287) > [info] 2021-10-28 06:07:18.948 - stdout> at > org.apache.spark.SparkEnv$.create(SparkEnv.scala:336) > [info] 2021-10-28 06:07:18.948 - stdout> at > org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:191) > [info] 2021-10-28 06:07:18.948 - stdout> at > org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:277) > [info] 2021-10-28 06:07:18.948 - stdout> at > org.apache.spark.SparkContext.<init>(SparkContext.scala:460) > [info] 2021-10-28 06:07:18.948 - stdout> at > org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58) > [info] 2021-10-28 06:07:18.948 - stdout> at > java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > [info] 2021-10-28 06:07:18.948 - stdout> at > java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) > [info] 2021-10-28 06:07:18.948 - stdout> at > java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > [info] 2021-10-28 06:07:18.948 - stdout> at > java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) > [info] 2021-10-28 06:07:18.948 - stdout> at > java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480) > [info] 2021-10-28 06:07:18.948 - stdout> at > py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247) > [info] 2021-10-28 06:07:18.948 - stdout> at > py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) > [info] 2021-10-28 06:07:18.948 - stdout> at > py4j.Gateway.invoke(Gateway.java:238) > [info] 2021-10-28 06:07:18.948 - stdout> at > py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80) > [info] 2021-10-28 06:07:18.948 - stdout> at > py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69) > [info] 2021-10-28 06:07:18.948 - stdout> at > py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182) > [info] 2021-10-28 06:07:18.948 - stdout> at > py4j.ClientServerConnection.run(ClientServerConnection.java:106) > [info] 2021-10-28 06:07:18.948 - stdout> at > java.base/java.lang.Thread.run(Thread.java:833) > [info] 2021-10-28 06:07:18.948 - stdout> > [info] 2021-10-28 06:07:18.975 - stderr> 21/10/28 22:07:18 INFO > ShutdownHookManager: Shutdown hook called > [info] 2021-10-28 06:07:18.976 - stderr> 21/10/28 22:07:18 INFO > ShutdownHookManager: Deleting directory > /tmp/spark-5bd38557-2b71-43f1-80ee-3648a456b3fc > (SparkSubmitTestUtils.scala:96) > [info] org.scalatest.exceptions.TestFailedException: > [info] at > org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472) > [info] at > org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471) > [info] at > org.scalatest.funsuite.AnyFunSuite.newAssertionFailedException(AnyFunSuite.scala:1563) > [info] at org.scalatest.Assertions.fail(Assertions.scala:933) > [info] at org.scalatest.Assertions.fail$(Assertions.scala:929) > [info] at org.scalatest.funsuite.AnyFunSuite.fail(AnyFunSuite.scala:1563) > [info] at > org.apache.spark.deploy.SparkSubmitTestUtils.runSparkSubmit(SparkSubmitTestUtils.scala:96) > [info] at > org.apache.spark.deploy.SparkSubmitTestUtils.runSparkSubmit$(SparkSubmitTestUtils.scala:43) > [info] at > org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite.runSparkSubmit(HiveExternalCatalogVersionsSuite.scala:56) > [info] at > org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite.$anonfun$beforeAll$2(HiveExternalCatalogVersionsSuite.scala:223) > [info] at > org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite.$anonfun$beforeAll$2$adapted(HiveExternalCatalogVersionsSuite.scala:204) > [info] at > scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62) > [info] at > scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55) > [info] at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) > [info] at > org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite.beforeAll(HiveExternalCatalogVersionsSuite.scala:204) > [info] at > org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212) > [info] at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210) > [info] at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208) > [info] at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:62) > [info] at > org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:318) > [info] at > org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:513) > [info] at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:413) > [info] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > [info] at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) > [info] at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) > [info] at java.base/java.lang.Thread.run(Thread.java:833) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org