Hi all, I encountered an UnsupportedOperationException when using Java 21 and the BigQuery Storage Write API in a Beam pipeline by using ".withMethod(BigQueryIO.Write.Method.STORAGE_WRITE_API));"
Having read issue #28120[1] and understanding that Beam version 2.52.0 or later supports Java 21 as a runtime, I wonder why such an error happens. I found there are two workarounds, but the Storage Write API is a more preferable way to insert data into BigQuery, so I'd like to find a solution. 1. One workaround is to switch from Java 21 to Java 17(openjdk version "17.0.10" 2024-01-16). By changing the <maven.compiler.source> and <maven.compiler.target> in the pom.xml file (i.e., without modifying App.java itself), the pipeline successfully writes data to my destination table on BigQuery. It seems Java 17 and BigQuery Storage Write API works fine. 2. The other workaround is to change insert method. I tried the BigQuery legacy streaming API( https://cloud.google.com/bigquery/docs/streaming-data-into-bigquery ) instead of the Storage Write API. Even though I still used Java 21, when I changed my code to .withMethod(BigQueryIO.Write.Method.STREAMING_INSERTS));, I did not encounter the error. So I faced the error only when using Java 21 and BigQuery Storage Write API. I uploaded the code below to reproduce. Could you please inform me how to handle this issue? https://github.com/cloud-ace/min-reproduce My Environment - OS - Ubuntu 22.04 - Mac OS Sonoma(14.3.1) - beam 2.53.0, 2.54.0 - openjdk version "21.0.2" 2024-01-16 - maven 3.9.6 - DirectRunner Thanks, Kazuha [1]: https://github.com/apache/beam/issues/28120 Here is the detailed error message. org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.lang.UnsupportedOperationException: Cannot define class using reflection: Unable to make protected java.lang.Package java.lang.ClassLoader.getPackage(java.lang.String) accessible: module java.base does not "opens java.lang" to unnamed module @116d5dff Caused by: java.lang.UnsupportedOperationException: Cannot define class using reflection: Unable to make protected java.lang.Package java.lang.ClassLoader.getPackage(java.lang.String) accessible: module java.base does not "opens java.lang" to unnamed module @116d5dff at net.bytebuddy.dynamic.loading.ClassInjector$UsingReflection$Dispatcher$Initializable$Unavailable.defineClass (ClassInjector.java:472) at net.bytebuddy.dynamic.loading.ClassInjector$UsingReflection.injectRaw (ClassInjector.java:284) at net.bytebuddy.dynamic.loading.ClassInjector$AbstractBase.inject (ClassInjector.java:118) at net.bytebuddy.dynamic.loading.ClassLoadingStrategy$Default$InjectionDispatcher.load (ClassLoadingStrategy.java:241) at net.bytebuddy.dynamic.loading.ClassLoadingStrategy$Default.load (ClassLoadingStrategy.java:148) at net.bytebuddy.dynamic.TypeResolutionStrategy$Passive.initialize (TypeResolutionStrategy.java:101) at net.bytebuddy.dynamic.DynamicType$Default$Unloaded.load (DynamicType.java:6317) at org.apache.beam.sdk.schemas.utils.AutoValueUtils.createBuilderCreator (AutoValueUtils.java:247) at org.apache.beam.sdk.schemas.utils.AutoValueUtils.getBuilderCreator (AutoValueUtils.java:225) at org.apache.beam.sdk.schemas.AutoValueSchema.schemaTypeCreator (AutoValueSchema.java:122) at org.apache.beam.sdk.schemas.CachingFactory.create (CachingFactory.java:56) at org.apache.beam.sdk.schemas.FromRowUsingCreator.apply (FromRowUsingCreator.java:94) at org.apache.beam.sdk.schemas.FromRowUsingCreator.apply (FromRowUsingCreator.java:45) at org.apache.beam.sdk.schemas.SchemaCoder.decode (SchemaCoder.java:126) at org.apache.beam.sdk.coders.Coder.decode (Coder.java:159) at org.apache.beam.sdk.coders.KvCoder.decode (KvCoder.java:84) at org.apache.beam.sdk.coders.KvCoder.decode (KvCoder.java:37) at org.apache.beam.sdk.util.CoderUtils.decodeFromSafeStream (CoderUtils.java:142) at org.apache.beam.sdk.util.CoderUtils.decodeFromByteArray (CoderUtils.java:102) at org.apache.beam.sdk.util.CoderUtils.decodeFromByteArray (CoderUtils.java:96) at org.apache.beam.sdk.util.CoderUtils.clone (CoderUtils.java:168) at org.apache.beam.sdk.util.MutationDetectors$CodedValueMutationDetector.<init> (MutationDetectors.java:118) at org.apache.beam.sdk.util.MutationDetectors.forValueWithCoder (MutationDetectors.java:49) at org.apache.beam.runners.direct.ImmutabilityCheckingBundleFactory$ImmutabilityEnforcingBundle.add (ImmutabilityCheckingBundleFactory.java:115) at org.apache.beam.runners.direct.ParDoEvaluator$BundleOutputManager.output (ParDoEvaluator.java:305) at org.apache.beam.repackaged.direct_java.runners.core.SimpleDoFnRunner.outputWindowedValue (SimpleDoFnRunner.java:275) at org.apache.beam.repackaged.direct_java.runners.core.SimpleDoFnRunner.access$900 (SimpleDoFnRunner.java:85) at org.apache.beam.repackaged.direct_java.runners.core.SimpleDoFnRunner$DoFnProcessContext.output (SimpleDoFnRunner.java:423) at org.apache.beam.sdk.transforms.DoFnOutputReceivers$WindowedContextOutputReceiver.output (DoFnOutputReceivers.java:76) at org.apache.beam.sdk.io.gcp.bigquery.StorageApiConvertMessages$ConvertMessagesDoFn.processElement (StorageApiConvertMessages.java:161) ＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝ Cloud Ace, Inc. Engineering Division/System Development Section/Data Solution Department TANAKA Kazuha 田中 万葉 〒100-0004 東京都千代田区大手町1丁目7番2号 東京サンケイビル26階 TEL: 03-6280-5939 FAX: 03-6800-3954 Email: tanaka.kaz...@cloud-ace.jp URL: https://www.cloud-ace.jp/ ＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝
