wattache commented on issue #34000: URL: https://github.com/apache/beam/issues/34000#issuecomment-3001020034
Hello @xuanswe From what I understood your question was about PubSub and not just handling failed inserts in BigQuery ? I have the same issue when dealing with a custom ClickHouse connector. As far as I understand the method [finalizeCheckpoint](https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubUnboundedSource.java#L267) is called. It seems to come from [Read.java](https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/Read.java#L612) when setting a bundle finalizer. So two observations: 1. it seems hard to change this behaviour with a quick win 2. the fact `bundleFinalizer` is called indeed suggests a fused step is made, when the fused step finished processing the bundle it's acked, and then any message between here and final destination (ClickHouse for me) is eventually lost -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
