Hey, can you provide the full stack trace for the error you're seeing? Also is this happening consistently?
*+1* to raising a Google ticket where we'll have more visibility. On Wed, Dec 6, 2023 at 11:33 AM John Casey <theotherj...@google.com> wrote: > Hmm. It may be best if you raise a ticket with Google support for this. I > can inspect your job directly if you do that, and that will make this more > straightforward. > > On Wed, Dec 6, 2023 at 11:24 AM hsy...@gmail.com <hsy...@gmail.com> wrote: > >> I’m just using dataflow engine >> On Wed, Dec 6, 2023 at 08:23 John Casey via user <user@beam.apache.org> >> wrote: >> >>> Well, that is odd. It looks like the underlying client is closed, which >>> is unexpected. >>> >>> Do you see any retries in your pipeline? Also, what runner are you using? >>> >>> @Ahmed Abualsaud <ahmedabuals...@google.com> this might be interesting >>> to you too >>> >>> On Tue, Dec 5, 2023 at 9:39 PM hsy...@gmail.com <hsy...@gmail.com> >>> wrote: >>> >>>> I'm using version 2.51.0 and The configuration is like this >>>> >>>> write >>>> .withoutValidation() >>>> .withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_NEVER) >>>> .withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_APPEND) >>>> .withExtendedErrorInfo() >>>> .withMethod(Write.Method.STORAGE_WRITE_API) >>>> .withTriggeringFrequency(Duration.standardSeconds(10)) >>>> .withAutoSharding().optimizedWrites() >>>> .withFailedInsertRetryPolicy(retryTransientErrors()); >>>> >>>> >>>> On Tue, Dec 5, 2023 at 11:20 AM John Casey via user < >>>> user@beam.apache.org> wrote: >>>> >>>>> Hi, >>>>> >>>>> Could you add some more detail? Which beam version are you using? >>>>> >>>>> >>>>> On Tue, Dec 5, 2023 at 1:52 PM hsy...@gmail.com <hsy...@gmail.com> >>>>> wrote: >>>>> >>>>>> Any one has experience in writing to BQ using storage api >>>>>> >>>>>> I tried to use it because according to the document it is more >>>>>> efficient >>>>>> but I got error below >>>>>> >>>>>> 2023-12-05 04:01:29.741 PST >>>>>> Error message from worker: java.lang.RuntimeException: >>>>>> java.lang.IllegalStateException >>>>>> org.apache.beam.sdk.io.gcp.bigquery.StorageApiWritesShardedRecords$WriteRecordsDoFn.lambda$process$12(StorageApiWritesShardedRecords.java:573) >>>>>> org.apache.beam.sdk.io.gcp.bigquery.StorageApiWritesShardedRecords$WriteRecordsDoFn.process(StorageApiWritesShardedRecords.java:790) >>>>>> Caused by: java.lang.IllegalStateException >>>>>> org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions.checkState(Preconditions.java:496) >>>>>> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl$1.pin(BigQueryServicesImpl.java:1403) >>>>>> org.apache.beam.sdk.io.gcp.bigquery.StorageApiWritesShardedRecords$WriteRecordsDoFn.lambda$process$12(StorageApiWritesShardedRecords.java:565) >>>>>> org.apache.beam.sdk.io.gcp.bigquery.StorageApiWritesShardedRecords$WriteRecordsDoFn.process(StorageApiWritesShardedRecords.java:790) >>>>>> org.apache.beam.sdk.io.gcp.bigquery.StorageApiWritesShardedRecords$WriteRecordsDoFn$DoFnInvoker.invokeProcessElement(Unknown >>>>>> Source) >>>>>> >>>>>
