Hi, This is Kensuke. I’m responding on behalf of Kazuha (he is my colleague).
> Which runner are you using? We are using the DirectRunner. As you suggested, we specified “–add-opens=java.base/java.lang=ALL-UNNAMED” in the JVM invocation command line, and it worked in the Java 21 environment. Specifically, after setting “export MAVEN_OPTS=‘–add-opens=java.base/java.lang=ALL-UNNAMED’”, we ran “mvn compile exec:java” and the error was resolved. > The same enforcement was introduced in both Java17 and 21, and it is > strange that Java17 worked without the option but Java21 didn't. I agree that it is strange. > Are you testing on the same beam version and other configurations? We have not made any changes other than what was mentioned in previous emails. > Try the latest beam version 2.56.0 and this option may not be needed. We tried using Beam version 2.56.0, but now we are encountering a different error and it doesn’t work. This error occurs with Beam 2.56.0 in both Java 17 and Java 21 environments. I have investigated the issue myself, but I couldn’t determine the cause. Do you have any insights based on this information? On 2024/06/07 14:18:07 Yi Hu via user wrote: > Hi, > > Which runner are you using? If you are running on Dataflow runner, then > refer to this [1] and add > "--jdkAddOpenModules=java.base/java.lang=ALL-UNNAMED" to pipeline option. > If using direct runner, then add > "--add-opens=java.base/java.lang=ALL-UNNAMED" to JVM invocation command > line. > > The same enforcement was introduced in both Java17 and 21, and it is > strange that Java17 worked without the option but Java21 didn't. Are you > testing on the same beam version and other configurations? Also, more > recent beam versions eliminated most usage of > "ClassLoadingStrategy.Default.INJECTION" > that cause this pipeline option being required, e.g. [2]. Try the latest > beam version 2.56.0 and this option may not be needed. > > [1] > https://beam.apache.org/releases/javadoc/current/org/apache/beam/runners/dataflow/options/DataflowPipelineOptions.html#getJdkAddOpenModules-- > > [2] https://github.com/apache/beam/pull/30367 > > > > On Mon, Jun 3, 2024 at 7:14 PM XQ Hu <xq...@google.com> wrote: > > > Probably related to the strict encapsulation that is enforced with Java > > 21. > > Use `--add-opens=java.base/java.lang=ALL-UNNAMED` as the JVM flag could be > > a temporary workaround. > > > > On Mon, Jun 3, 2024 at 3:04 AM 田中万葉 <ta...@cloud-ace.jp> wrote: > > > >> Hi all, > >> > >> I encountered an UnsupportedOperationException when using Java 21 and the > >> BigQuery Storage Write API in a Beam pipeline by using > >> ".withMethod(BigQueryIO.Write.Method.STORAGE_WRITE_API));" > >> > >> Having read issue #28120[1] and understanding that Beam version 2.52.0 or > >> later supports Java 21 as a runtime, I wonder why such an error happens. > >> > >> I found there are two workarounds, but the Storage Write API is a more > >> preferable way to insert data into BigQuery, so I'd like to find a > >> solution. > >> > >> 1. One workaround is to switch from Java 21 to Java 17(openjdk version > >> "17.0.10" 2024-01-16). By changing the <maven.compiler.source> and > >> <maven.compiler.target> in the pom.xml file (i.e., without modifying > >> App.java itself), the pipeline successfully writes data to my destination > >> table on BigQuery. It seems Java 17 and BigQuery Storage Write API works > >> fine. > >> 2. The other workaround is to change insert method. I tried the BigQuery > >> legacy streaming API( > >> https://cloud.google.com/bigquery/docs/streaming-data-into-bigquery ) > >> instead of the Storage Write API. Even though I still used Java 21, when I > >> changed my code to > >> .withMethod(BigQueryIO.Write.Method.STREAMING_INSERTS));, I did not > >> encounter the error. > >> > >> So I faced the error only when using Java 21 and BigQuery Storage Write > >> API. > >> > >> I uploaded the code below to reproduce. Could you please inform me how to > >> handle this issue? > >> https://github.com/cloud-ace/min-reproduce > >> > >> My Environment > >> - OS > >> - Ubuntu 22.04 > >> - Mac OS Sonoma(14.3.1) > >> - beam 2.53.0, 2.54.0 > >> - openjdk version "21.0.2" 2024-01-16 > >> - maven 3.9.6 > >> - DirectRunner > >> > >> Thanks, > >> > >> Kazuha > >> > >> [1]: https://github.com/apache/beam/issues/28120 > >> > >> Here is the detailed error message. > >> > >> org.apache.beam.sdk.Pipeline$PipelineExecutionException: > >> java.lang.UnsupportedOperationException: Cannot define class using > >> reflection: Unable to make protected java.lang.Package > >> java.lang.ClassLoader.getPackage(java.lang.String) accessible: module > >> java.base does not "opens java.lang" to unnamed module @116d5dff > >> > >> Caused by: java.lang.UnsupportedOperationException: Cannot define class > >> using reflection: Unable to make protected java.lang.Package > >> java.lang.ClassLoader.getPackage(java.lang.String) accessible: module > >> java.base does not "opens java.lang" to unnamed module @116d5dff > >> at > >> net.bytebuddy.dynamic.loading.ClassInjector$UsingReflection$Dispatcher$Initializable$Unavailable.defineClass > >> (ClassInjector.java:472) > >> at > >> net.bytebuddy.dynamic.loading.ClassInjector$UsingReflection.injectRaw > >> (ClassInjector.java:284) > >> at net.bytebuddy.dynamic.loading.ClassInjector$AbstractBase.inject > >> (ClassInjector.java:118) > >> at > >> net.bytebuddy.dynamic.loading.ClassLoadingStrategy$Default$InjectionDispatcher.load > >> (ClassLoadingStrategy.java:241) > >> at net.bytebuddy.dynamic.loading.ClassLoadingStrategy$Default.load > >> (ClassLoadingStrategy.java:148) > >> at net.bytebuddy.dynamic.TypeResolutionStrategy$Passive.initialize > >> (TypeResolutionStrategy.java:101) > >> at net.bytebuddy.dynamic.DynamicType$Default$Unloaded.load > >> (DynamicType.java:6317) > >> at > >> org.apache.beam.sdk.schemas.utils.AutoValueUtils.createBuilderCreator > >> (AutoValueUtils.java:247) > >> at org.apache.beam.sdk.schemas.utils.AutoValueUtils.getBuilderCreator > >> (AutoValueUtils.java:225) > >> at org.apache.beam.sdk.schemas.AutoValueSchema.schemaTypeCreator > >> (AutoValueSchema.java:122) > >> at org.apache.beam.sdk.schemas.CachingFactory.create > >> (CachingFactory.java:56) > >> at org.apache.beam.sdk.schemas.FromRowUsingCreator.apply > >> (FromRowUsingCreator.java:94) > >> at org.apache.beam.sdk.schemas.FromRowUsingCreator.apply > >> (FromRowUsingCreator.java:45) > >> at org.apache.beam.sdk.schemas.SchemaCoder.decode > >> (SchemaCoder.java:126) > >> at org.apache.beam.sdk.coders.Coder.decode (Coder.java:159) > >> at org.apache.beam.sdk.coders.KvCoder.decode (KvCoder.java:84) > >> at org.apache.beam.sdk.coders.KvCoder.decode (KvCoder.java:37) > >> at org.apache.beam.sdk.util.CoderUtils.decodeFromSafeStream > >> (CoderUtils.java:142) > >> at org.apache.beam.sdk.util.CoderUtils.decodeFromByteArray > >> (CoderUtils.java:102) > >> at org.apache.beam.sdk.util.CoderUtils.decodeFromByteArray > >> (CoderUtils.java:96) > >> at org.apache.beam.sdk.util.CoderUtils.clone (CoderUtils.java:168) > >> at > >> org.apache.beam.sdk.util.MutationDetectors$CodedValueMutationDetector.<init> > >> (MutationDetectors.java:118) > >> at org.apache.beam.sdk.util.MutationDetectors.forValueWithCoder > >> (MutationDetectors.java:49) > >> at > >> org.apache.beam.runners.direct.ImmutabilityCheckingBundleFactory$ImmutabilityEnforcingBundle.add > >> (ImmutabilityCheckingBundleFactory.java:115) > >> at > >> org.apache.beam.runners.direct.ParDoEvaluator$BundleOutputManager.output > >> (ParDoEvaluator.java:305) > >> at > >> org.apache.beam.repackaged.direct_java.runners.core.SimpleDoFnRunner.outputWindowedValue > >> (SimpleDoFnRunner.java:275) > >> at > >> org.apache.beam.repackaged.direct_java.runners.core.SimpleDoFnRunner.access$900 > >> (SimpleDoFnRunner.java:85) > >> at > >> org.apache.beam.repackaged.direct_java.runners.core.SimpleDoFnRunner$DoFnProcessContext.output > >> (SimpleDoFnRunner.java:423) > >> at > >> org.apache.beam.sdk.transforms.DoFnOutputReceivers$WindowedContextOutputReceiver.output > >> (DoFnOutputReceivers.java:76) > >> at > >> org.apache.beam.sdk.io.gcp.bigquery.StorageApiConvertMessages$ConvertMessagesDoFn.processElement > >> (StorageApiConvertMessages.java:161) > >> > >> ＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝ > >> Cloud Ace, Inc. > >> Engineering Division/System Development Section/Data Solution Department > >> TANAKA Kazuha 田中 万葉 > >> > >> 〒100-0004 東京都千代田区大手町1丁目7番2号 東京サンケイビル26階 > >> > >> TEL: 03-6280-5939 FAX: 03-6800-3954 > >> Email: tanaka.kaz...@cloud-ace.jp > >> URL: https://www.cloud-ace.jp/ > >> ＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝ > >> > > > > -- > > Yi Hu, (he/him/his) > > Software Engineer >
