Hi, Which runner are you using? If you are running on Dataflow runner, then refer to this [1] and add "--jdkAddOpenModules=java.base/java.lang=ALL-UNNAMED" to pipeline option. If using direct runner, then add "--add-opens=java.base/java.lang=ALL-UNNAMED" to JVM invocation command line.
The same enforcement was introduced in both Java17 and 21, and it is strange that Java17 worked without the option but Java21 didn't. Are you testing on the same beam version and other configurations? Also, more recent beam versions eliminated most usage of "ClassLoadingStrategy.Default.INJECTION" that cause this pipeline option being required, e.g. [2]. Try the latest beam version 2.56.0 and this option may not be needed. [1] https://beam.apache.org/releases/javadoc/current/org/apache/beam/runners/dataflow/options/DataflowPipelineOptions.html#getJdkAddOpenModules-- [2] https://github.com/apache/beam/pull/30367 On Mon, Jun 3, 2024 at 7:14 PM XQ Hu <x...@google.com> wrote: > Probably related to the strict encapsulation that is enforced with Java > 21. > Use `--add-opens=java.base/java.lang=ALL-UNNAMED` as the JVM flag could be > a temporary workaround. > > On Mon, Jun 3, 2024 at 3:04 AM 田中万葉 <tanaka.kaz...@cloud-ace.jp> wrote: > >> Hi all, >> >> I encountered an UnsupportedOperationException when using Java 21 and the >> BigQuery Storage Write API in a Beam pipeline by using >> ".withMethod(BigQueryIO.Write.Method.STORAGE_WRITE_API));" >> >> Having read issue #28120[1] and understanding that Beam version 2.52.0 or >> later supports Java 21 as a runtime, I wonder why such an error happens. >> >> I found there are two workarounds, but the Storage Write API is a more >> preferable way to insert data into BigQuery, so I'd like to find a >> solution. >> >> 1. One workaround is to switch from Java 21 to Java 17(openjdk version >> "17.0.10" 2024-01-16). By changing the <maven.compiler.source> and >> <maven.compiler.target> in the pom.xml file (i.e., without modifying >> App.java itself), the pipeline successfully writes data to my destination >> table on BigQuery. It seems Java 17 and BigQuery Storage Write API works >> fine. >> 2. The other workaround is to change insert method. I tried the BigQuery >> legacy streaming API( >> https://cloud.google.com/bigquery/docs/streaming-data-into-bigquery ) >> instead of the Storage Write API. Even though I still used Java 21, when I >> changed my code to >> .withMethod(BigQueryIO.Write.Method.STREAMING_INSERTS));, I did not >> encounter the error. >> >> So I faced the error only when using Java 21 and BigQuery Storage Write >> API. >> >> I uploaded the code below to reproduce. Could you please inform me how to >> handle this issue? >> https://github.com/cloud-ace/min-reproduce >> >> My Environment >> - OS >> - Ubuntu 22.04 >> - Mac OS Sonoma(14.3.1) >> - beam 2.53.0, 2.54.0 >> - openjdk version "21.0.2" 2024-01-16 >> - maven 3.9.6 >> - DirectRunner >> >> Thanks, >> >> Kazuha >> >> [1]: https://github.com/apache/beam/issues/28120 >> >> Here is the detailed error message. >> >> org.apache.beam.sdk.Pipeline$PipelineExecutionException: >> java.lang.UnsupportedOperationException: Cannot define class using >> reflection: Unable to make protected java.lang.Package >> java.lang.ClassLoader.getPackage(java.lang.String) accessible: module >> java.base does not "opens java.lang" to unnamed module @116d5dff >> >> Caused by: java.lang.UnsupportedOperationException: Cannot define class >> using reflection: Unable to make protected java.lang.Package >> java.lang.ClassLoader.getPackage(java.lang.String) accessible: module >> java.base does not "opens java.lang" to unnamed module @116d5dff >> at >> net.bytebuddy.dynamic.loading.ClassInjector$UsingReflection$Dispatcher$Initializable$Unavailable.defineClass >> (ClassInjector.java:472) >> at >> net.bytebuddy.dynamic.loading.ClassInjector$UsingReflection.injectRaw >> (ClassInjector.java:284) >> at net.bytebuddy.dynamic.loading.ClassInjector$AbstractBase.inject >> (ClassInjector.java:118) >> at >> net.bytebuddy.dynamic.loading.ClassLoadingStrategy$Default$InjectionDispatcher.load >> (ClassLoadingStrategy.java:241) >> at net.bytebuddy.dynamic.loading.ClassLoadingStrategy$Default.load >> (ClassLoadingStrategy.java:148) >> at net.bytebuddy.dynamic.TypeResolutionStrategy$Passive.initialize >> (TypeResolutionStrategy.java:101) >> at net.bytebuddy.dynamic.DynamicType$Default$Unloaded.load >> (DynamicType.java:6317) >> at >> org.apache.beam.sdk.schemas.utils.AutoValueUtils.createBuilderCreator >> (AutoValueUtils.java:247) >> at org.apache.beam.sdk.schemas.utils.AutoValueUtils.getBuilderCreator >> (AutoValueUtils.java:225) >> at org.apache.beam.sdk.schemas.AutoValueSchema.schemaTypeCreator >> (AutoValueSchema.java:122) >> at org.apache.beam.sdk.schemas.CachingFactory.create >> (CachingFactory.java:56) >> at org.apache.beam.sdk.schemas.FromRowUsingCreator.apply >> (FromRowUsingCreator.java:94) >> at org.apache.beam.sdk.schemas.FromRowUsingCreator.apply >> (FromRowUsingCreator.java:45) >> at org.apache.beam.sdk.schemas.SchemaCoder.decode >> (SchemaCoder.java:126) >> at org.apache.beam.sdk.coders.Coder.decode (Coder.java:159) >> at org.apache.beam.sdk.coders.KvCoder.decode (KvCoder.java:84) >> at org.apache.beam.sdk.coders.KvCoder.decode (KvCoder.java:37) >> at org.apache.beam.sdk.util.CoderUtils.decodeFromSafeStream >> (CoderUtils.java:142) >> at org.apache.beam.sdk.util.CoderUtils.decodeFromByteArray >> (CoderUtils.java:102) >> at org.apache.beam.sdk.util.CoderUtils.decodeFromByteArray >> (CoderUtils.java:96) >> at org.apache.beam.sdk.util.CoderUtils.clone (CoderUtils.java:168) >> at >> org.apache.beam.sdk.util.MutationDetectors$CodedValueMutationDetector.<init> >> (MutationDetectors.java:118) >> at org.apache.beam.sdk.util.MutationDetectors.forValueWithCoder >> (MutationDetectors.java:49) >> at >> org.apache.beam.runners.direct.ImmutabilityCheckingBundleFactory$ImmutabilityEnforcingBundle.add >> (ImmutabilityCheckingBundleFactory.java:115) >> at >> org.apache.beam.runners.direct.ParDoEvaluator$BundleOutputManager.output >> (ParDoEvaluator.java:305) >> at >> org.apache.beam.repackaged.direct_java.runners.core.SimpleDoFnRunner.outputWindowedValue >> (SimpleDoFnRunner.java:275) >> at >> org.apache.beam.repackaged.direct_java.runners.core.SimpleDoFnRunner.access$900 >> (SimpleDoFnRunner.java:85) >> at >> org.apache.beam.repackaged.direct_java.runners.core.SimpleDoFnRunner$DoFnProcessContext.output >> (SimpleDoFnRunner.java:423) >> at >> org.apache.beam.sdk.transforms.DoFnOutputReceivers$WindowedContextOutputReceiver.output >> (DoFnOutputReceivers.java:76) >> at >> org.apache.beam.sdk.io.gcp.bigquery.StorageApiConvertMessages$ConvertMessagesDoFn.processElement >> (StorageApiConvertMessages.java:161) >> >> ＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝ >> Cloud Ace, Inc. >> Engineering Division/System Development Section/Data Solution Department >> TANAKA Kazuha 田中 万葉 >> >> 〒100-0004 東京都千代田区大手町1丁目7番2号 東京サンケイビル26階 >> >> TEL: 03-6280-5939 FAX: 03-6800-3954 >> Email: tanaka.kaz...@cloud-ace.jp >> URL: https://www.cloud-ace.jp/ >> ＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝ >> > -- Yi Hu, (he/him/his) Software Engineer
