userzhy opened a new pull request, #37178: URL: https://github.com/apache/beam/pull/37178
Fix PubsubUnboundedSink's WriterFn not enforcing message count limit per publish batch ### Description This PR fixes a bug where `PubsubUnboundedSink`'s `WriterFn` was not properly enforcing the message count limit per publish batch. The `publishBatchSize` parameter was being passed to the constructor but was never stored or used in the `processElement` method. According to [Google Cloud Pub/Sub resource limits](https://cloud.google.com/pubsub/quotas#resource_limits), a single publish request can contain at most 1000 messages. Without this fix, batches could exceed this limit, causing Pub/Sub to reject the publish request. ### Changes - Add `publishBatchSize` field to `WriterFn` class - Store `publishBatchSize` in both `WriterFn` constructors - Add message count check in `processElement` alongside existing byte size check - Update both `PubsubSinkDynamicDestinations` and `PubsubSink` to pass `publishBatchSize` when creating `WriterFn` instances ### Testing - Existing `PubsubUnboundedSinkTest` tests pass - The test `sendMoreThanOneBatchByNumMessages` specifically validates batch splitting by message count Fixes #36885 ------------------------ Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [x] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/CONTRIBUTING.md#make-the-reviewers-job-easier). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
