Probably related to the strict encapsulation that is enforced with Java 21. Use `--add-opens=java.base/java.lang=ALL-UNNAMED` as the JVM flag could be a temporary workaround.
On Mon, Jun 3, 2024 at 3:04 AM 田中万葉 <tanaka.kaz...@cloud-ace.jp> wrote: > Hi all, > > I encountered an UnsupportedOperationException when using Java 21 and the > BigQuery Storage Write API in a Beam pipeline by using > ".withMethod(BigQueryIO.Write.Method.STORAGE_WRITE_API));" > > Having read issue #28120[1] and understanding that Beam version 2.52.0 or > later supports Java 21 as a runtime, I wonder why such an error happens. > > I found there are two workarounds, but the Storage Write API is a more > preferable way to insert data into BigQuery, so I'd like to find a > solution. > > 1. One workaround is to switch from Java 21 to Java 17(openjdk version > "17.0.10" 2024-01-16). By changing the <maven.compiler.source> and > <maven.compiler.target> in the pom.xml file (i.e., without modifying > App.java itself), the pipeline successfully writes data to my destination > table on BigQuery. It seems Java 17 and BigQuery Storage Write API works > fine. > 2. The other workaround is to change insert method. I tried the BigQuery > legacy streaming API( > https://cloud.google.com/bigquery/docs/streaming-data-into-bigquery ) > instead of the Storage Write API. Even though I still used Java 21, when I > changed my code to > .withMethod(BigQueryIO.Write.Method.STREAMING_INSERTS));, I did not > encounter the error. > > So I faced the error only when using Java 21 and BigQuery Storage Write > API. > > I uploaded the code below to reproduce. Could you please inform me how to > handle this issue? > https://github.com/cloud-ace/min-reproduce > > My Environment > - OS > - Ubuntu 22.04 > - Mac OS Sonoma(14.3.1) > - beam 2.53.0, 2.54.0 > - openjdk version "21.0.2" 2024-01-16 > - maven 3.9.6 > - DirectRunner > > Thanks, > > Kazuha > > [1]: https://github.com/apache/beam/issues/28120 > > Here is the detailed error message. > > org.apache.beam.sdk.Pipeline$PipelineExecutionException: > java.lang.UnsupportedOperationException: Cannot define class using > reflection: Unable to make protected java.lang.Package > java.lang.ClassLoader.getPackage(java.lang.String) accessible: module > java.base does not "opens java.lang" to unnamed module @116d5dff > > Caused by: java.lang.UnsupportedOperationException: Cannot define class > using reflection: Unable to make protected java.lang.Package > java.lang.ClassLoader.getPackage(java.lang.String) accessible: module > java.base does not "opens java.lang" to unnamed module @116d5dff > at > net.bytebuddy.dynamic.loading.ClassInjector$UsingReflection$Dispatcher$Initializable$Unavailable.defineClass > (ClassInjector.java:472) > at > net.bytebuddy.dynamic.loading.ClassInjector$UsingReflection.injectRaw > (ClassInjector.java:284) > at net.bytebuddy.dynamic.loading.ClassInjector$AbstractBase.inject > (ClassInjector.java:118) > at > net.bytebuddy.dynamic.loading.ClassLoadingStrategy$Default$InjectionDispatcher.load > (ClassLoadingStrategy.java:241) > at net.bytebuddy.dynamic.loading.ClassLoadingStrategy$Default.load > (ClassLoadingStrategy.java:148) > at net.bytebuddy.dynamic.TypeResolutionStrategy$Passive.initialize > (TypeResolutionStrategy.java:101) > at net.bytebuddy.dynamic.DynamicType$Default$Unloaded.load > (DynamicType.java:6317) > at > org.apache.beam.sdk.schemas.utils.AutoValueUtils.createBuilderCreator > (AutoValueUtils.java:247) > at org.apache.beam.sdk.schemas.utils.AutoValueUtils.getBuilderCreator > (AutoValueUtils.java:225) > at org.apache.beam.sdk.schemas.AutoValueSchema.schemaTypeCreator > (AutoValueSchema.java:122) > at org.apache.beam.sdk.schemas.CachingFactory.create > (CachingFactory.java:56) > at org.apache.beam.sdk.schemas.FromRowUsingCreator.apply > (FromRowUsingCreator.java:94) > at org.apache.beam.sdk.schemas.FromRowUsingCreator.apply > (FromRowUsingCreator.java:45) > at org.apache.beam.sdk.schemas.SchemaCoder.decode > (SchemaCoder.java:126) > at org.apache.beam.sdk.coders.Coder.decode (Coder.java:159) > at org.apache.beam.sdk.coders.KvCoder.decode (KvCoder.java:84) > at org.apache.beam.sdk.coders.KvCoder.decode (KvCoder.java:37) > at org.apache.beam.sdk.util.CoderUtils.decodeFromSafeStream > (CoderUtils.java:142) > at org.apache.beam.sdk.util.CoderUtils.decodeFromByteArray > (CoderUtils.java:102) > at org.apache.beam.sdk.util.CoderUtils.decodeFromByteArray > (CoderUtils.java:96) > at org.apache.beam.sdk.util.CoderUtils.clone (CoderUtils.java:168) > at > org.apache.beam.sdk.util.MutationDetectors$CodedValueMutationDetector.<init> > (MutationDetectors.java:118) > at org.apache.beam.sdk.util.MutationDetectors.forValueWithCoder > (MutationDetectors.java:49) > at > org.apache.beam.runners.direct.ImmutabilityCheckingBundleFactory$ImmutabilityEnforcingBundle.add > (ImmutabilityCheckingBundleFactory.java:115) > at > org.apache.beam.runners.direct.ParDoEvaluator$BundleOutputManager.output > (ParDoEvaluator.java:305) > at > org.apache.beam.repackaged.direct_java.runners.core.SimpleDoFnRunner.outputWindowedValue > (SimpleDoFnRunner.java:275) > at > org.apache.beam.repackaged.direct_java.runners.core.SimpleDoFnRunner.access$900 > (SimpleDoFnRunner.java:85) > at > org.apache.beam.repackaged.direct_java.runners.core.SimpleDoFnRunner$DoFnProcessContext.output > (SimpleDoFnRunner.java:423) > at > org.apache.beam.sdk.transforms.DoFnOutputReceivers$WindowedContextOutputReceiver.output > (DoFnOutputReceivers.java:76) > at > org.apache.beam.sdk.io.gcp.bigquery.StorageApiConvertMessages$ConvertMessagesDoFn.processElement > (StorageApiConvertMessages.java:161) > > ＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝ > Cloud Ace, Inc. > Engineering Division/System Development Section/Data Solution Department > TANAKA Kazuha 田中 万葉 > > 〒100-0004 東京都千代田区大手町1丁目7番2号 東京サンケイビル26階 > > TEL: 03-6280-5939 FAX: 03-6800-3954 > Email: tanaka.kaz...@cloud-ace.jp > URL: https://www.cloud-ace.jp/ > ＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝ >
