| Hi. I am running on macOS 12.4, using an ‘Adoptium’ JDK from https://adoptium.net/download. The version details are: $ java -version openjdk version "17.0.3" 2022-04-19 OpenJDK Runtime Environment Temurin-17.0.3+7 (build 17.0.3+7) OpenJDK 64-Bit Server VM Temurin-17.0.3+7 (build 17.0.3+7, mixed mode, sharing) I have attached an example maven project which demonstrates the error. |
spark-java-17.tar.bz2
Description: BZip2 compressed data
If you run 'mvn clean test' it should fail with: [ERROR] ExampleTest Time elapsed: 1.194 s <<< ERROR! java.lang.IllegalAccessError: class org.apache.spark.storage.StorageUtils$ (in unnamed module @0x41a962cf) cannot access class sun.nio.ch.DirectBuffer (in module java.base) because module java.base does not export sun.nio.ch to unnamed module @0x41a962cf Some of the diagnostic output from running with Maven with the -X flag is: Apache Maven 3.8.6 (84538c9988a25aec085021c365c560670ad80f63) Maven home: /usr/local/apache-maven/apache-maven-3.8.6 Java version: 17.0.3, vendor: Eclipse Adoptium, runtime: /Library/Java/JavaVirtualMachines/temurin-17.jdk/Contents/Home Default locale: en_AU, platform encoding: UTF-8 OS name: "mac os x", version: "12.4", arch: "x86_64", family: “mac" [DEBUG] boot(compact) classpath: surefire-booter-3.0.0-M7.jar surefire-api-3.0.0-M7.jar surefire-logger-api-3.0.0-M7.jar surefire-shared-utils-3.0.0-M7.jar surefire-extensions-spi-3.0.0-M7.jar test-classes classes junit-4.13.2.jar hamcrest-core-1.3.jar hamcrest-all-1.3.jar spark-core_2.12-3.3.0.jar avro-1.11.0.jar jackson-core-2.12.5.jar commons-compress-1.21.jar avro-mapred-1.11.0.jar avro-ipc-1.11.0.jar xz-1.9.jar chill_2.12-0.10.0.jar kryo-shaded-4.0.2.jar minlog-1.3.0.jar objenesis-2.5.1.jar chill-java-0.10.0.jar xbean-asm9-shaded-4.20.jar hadoop-client-api-3.3.2.jar hadoop-client-runtime-3.3.2.jar commons-logging-1.1.3.jar spark-launcher_2.12-3.3.0.jar spark-kvstore_2.12-3.3.0.jar leveldbjni-all-1.8.jar jackson-annotations-2.13.3.jar spark-network-common_2.12-3.3.0.jar tink-1.6.1.jar gson-2.8.6.jar spark-network-shuffle_2.12-3.3.0.jar spark-unsafe_2.12-3.3.0.jar activation-1.1.1.jar curator-recipes-2.13.0.jar curator-framework-2.13.0.jar curator-client-2.13.0.jar guava-16.0.1.jar zookeeper-3.6.2.jar commons-lang-2.6.jar zookeeper-jute-3.6.2.jar audience-annotations-0.5.0.jar jakarta.servlet-api-4.0.3.jar commons-codec-1.15.jar commons-lang3-3.12.0.jar commons-math3-3.6.1.jar commons-text-1.9.jar commons-io-2.11.0.jar commons-collections-3.2.2.jar commons-collections4-4.4.jar jsr305-3.0.0.jar slf4j-api-1.7.32.jar jul-to-slf4j-1.7.32.jar jcl-over-slf4j-1.7.32.jar log4j-slf4j-impl-2.17.2.jar log4j-api-2.17.2.jar log4j-core-2.17.2.jar log4j-1.2-api-2.17.2.jar compress-lzf-1.1.jar snappy-java-1.1.8.4.jar lz4-java-1.8.0.jar zstd-jni-1.5.2-1.jar RoaringBitmap-0.9.25.jar shims-0.9.25.jar scala-xml_2.12-1.2.0.jar scala-library-2.12.15.jar scala-reflect-2.12.15.jar json4s-jackson_2.12-3.7.0-M11.jar json4s-core_2.12-3.7.0-M11.jar json4s-ast_2.12-3.7.0-M11.jar json4s-scalap_2.12-3.7.0-M11.jar jersey-client-2.34.jar jakarta.ws.rs-api-2.1.6.jar jakarta.inject-2.6.1.jar jersey-common-2.34.jar jakarta.annotation-api-1.3.5.jar osgi-resource-locator-1.0.3.jar jersey-server-2.34.jar jakarta.validation-api-2.0.2.jar jersey-container-servlet-2.34.jar jersey-container-servlet-core-2.34.jar jersey-hk2-2.34.jar hk2-locator-2.6.1.jar aopalliance-repackaged-2.6.1.jar hk2-api-2.6.1.jar hk2-utils-2.6.1.jar javassist-3.25.0-GA.jar netty-all-4.1.74.Final.jar netty-buffer-4.1.74.Final.jar netty-codec-4.1.74.Final.jar netty-common-4.1.74.Final.jar netty-handler-4.1.74.Final.jar netty-tcnative-classes-2.0.48.Final.jar netty-resolver-4.1.74.Final.jar netty-transport-4.1.74.Final.jar netty-transport-classes-epoll-4.1.74.Final.jar netty-transport-native-unix-common-4.1.74.Final.jar netty-transport-classes-kqueue-4.1.74.Final.jar netty-transport-native-epoll-4.1.74.Final-linux-x86_64.jar netty-transport-native-epoll-4.1.74.Final-linux-aarch_64.jar netty-transport-native-kqueue-4.1.74.Final-osx-x86_64.jar netty-transport-native-kqueue-4.1.74.Final-osx-aarch_64.jar stream-2.9.6.jar metrics-core-4.2.7.jar metrics-jvm-4.2.7.jar metrics-json-4.2.7.jar metrics-graphite-4.2.7.jar metrics-jmx-4.2.7.jar jackson-databind-2.13.3.jar jackson-module-scala_2.12-2.13.3.jar paranamer-2.8.jar ivy-2.5.0.jar oro-2.0.8.jar pickle-1.2.jar py4j-0.10.9.5.jar spark-tags_2.12-3.3.0.jar commons-crypto-1.1.0.jar unused-1.0.0.jar spark-sql_2.12-3.3.0.jar rocksdbjni-6.20.3.jar univocity-parsers-2.9.1.jar spark-sketch_2.12-3.3.0.jar spark-catalyst_2.12-3.3.0.jar scala-parser-combinators_2.12-1.1.2.jar janino-3.0.16.jar commons-compiler-3.0.16.jar antlr4-runtime-4.8.jar arrow-vector-7.0.0.jar arrow-format-7.0.0.jar arrow-memory-core-7.0.0.jar flatbuffers-java-1.12.0.jar arrow-memory-netty-7.0.0.jar orc-core-1.7.4.jar orc-shims-1.7.4.jar protobuf-java-2.5.0.jar aircompressor-0.21.jar annotations-17.0.0.jar threeten-extra-1.5.0.jar orc-mapreduce-1.7.4.jar hive-storage-api-2.7.2.jar parquet-column-1.12.2.jar parquet-common-1.12.2.jar parquet-encoding-1.12.2.jar parquet-hadoop-1.12.2.jar parquet-format-structures-1.12.2.jar parquet-jackson-1.12.2.jar surefire-junit4-3.0.0-M7.jar common-java5-3.0.0-M7.jar common-junit3-3.0.0-M7.jar common-junit4-3.0.0-M7.jar [DEBUG] Forking command line: /bin/sh -c cd '/Users/greg/devel/spark-java17' && '/Library/Java/JavaVirtualMachines/temurin-17.jdk/Contents/Home/bin/java' '-jar' '/Users/greg/devel/spark-java17/target/surefire/surefirebooter-20220623160237651_3.jar' '/Users/greg/devel/spark-java17/target/surefire' '2022-06-23T16-02-37_489-jvmRun1' 'surefire-20220623160237651_1tmp' 'surefire_0-20220623160237651_2tmp' [DEBUG] Fork Channel [1] connected to the client. [INFO] Running ExampleTest Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties 22/06/23 16:02:38 INFO SparkContext: Running Spark version 3.3.0 22/06/23 16:02:38 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 22/06/23 16:02:38 INFO ResourceUtils: ============================================================== 22/06/23 16:02:38 INFO ResourceUtils: No custom resources configured for spark.driver. 22/06/23 16:02:38 INFO ResourceUtils: ============================================================== 22/06/23 16:02:38 INFO SparkContext: Submitted application: e42fa5ea-9b36-42e9-86fe-0a8d58f451b8 22/06/23 16:02:38 INFO ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 1024, script: , vendor: , offHeap -> name: offHeap, amount: 0, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0) 22/06/23 16:02:38 INFO ResourceProfile: Limiting resource is cpu 22/06/23 16:02:38 INFO ResourceProfileManager: Added ResourceProfile id: 0 22/06/23 16:02:38 INFO SecurityManager: Changing view acls to: greg 22/06/23 16:02:38 INFO SecurityManager: Changing modify acls to: greg 22/06/23 16:02:38 INFO SecurityManager: Changing view acls groups to: 22/06/23 16:02:38 INFO SecurityManager: Changing modify acls groups to: 22/06/23 16:02:38 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(greg); groups with view permissions: Set(); users with modify permissions: Set(greg); groups with modify permissions: Set() 22/06/23 16:02:39 INFO Utils: Successfully started service 'sparkDriver' on port 56377. 22/06/23 16:02:39 INFO SparkEnv: Registering MapOutputTracker 22/06/23 16:02:39 INFO SparkEnv: Registering BlockManagerMaster 22/06/23 16:02:39 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information 22/06/23 16:02:39 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up [ERROR] Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 1.2 s <<< FAILURE! - in ExampleTest [ERROR] ExampleTest Time elapsed: 1.194 s <<< ERROR! java.lang.IllegalAccessError: class org.apache.spark.storage.StorageUtils$ (in unnamed module @0x41a962cf) cannot access class sun.nio.ch.DirectBuffer (in module java.base) because module java.base does not export sun.nio.ch to unnamed module @0x41a962cf at org.apache.spark.storage.StorageUtils$.<init>(StorageUtils.scala:213) If I change: <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-surefire-plugin</artifactId> <version>3.0.0-M7</version> </plugin> </plugins> to: <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-surefire-plugin</artifactId> <version>3.0.0-M7</version> <configuration> <argLine>--add-exports java.base/sun.nio.ch=ALL-UNNAMED</argLine> </configuration> </plugin> </plugins> … then the tests run successfully. If there’s anything else you need, please let me know. — Greg.
|
