1fanwang opened a new pull request, #39886: URL: https://github.com/apache/beam/pull/39886
Bounded PubsubIO writes size each batch from the message payload alone. Attributes and ordering keys are omitted, so the writer can publish a batch larger than `maxPublishBatchByteSize`. `validatePubsubMessage` already returns the validated size of the payload, ordering key, and attributes. The bounded writer now uses that value instead of discarding it and recalculating `messageSize` from the payload. This does not address https://github.com/apache/beam/issues/31800. Its existing TODOs cover JSON versus Protobuf encoding and HTTP to gRPC transcoding. Fixes #28011 ------------------------ ## Testing The regression sends two 27-byte messages through the real bounded-writer lifecycle with a 40-byte batch limit and captures each publish call. To reproduce against base `13875fc6bd3d5ad534c89d33f2395176fbc485fc` while keeping the regression test from this branch: ```bash git checkout 13875fc6bd3d5ad534c89d33f2395176fbc485fc -- \ sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.java export JAVA_HOME=$(/usr/libexec/java_home -v 21.0.6) export PATH="$JAVA_HOME/bin:$PATH" ./gradlew :sdks:java:io:google-cloud-platform:test \ --tests org.apache.beam.sdk.io.gcp.pubsub.PubsubIOTest.testBoundedWriteBatchSizeIncludesAttributesAndOrderingKey \ --no-daemon --console=plain git checkout HEAD -- \ sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.java ``` Both messages were sent in one batch: ```text java.lang.AssertionError: expected:<[1, 1]> but was:<[2]> 1 test completed, 1 failed BUILD FAILED in 17s ``` After the fix: ```bash export JAVA_HOME=$(/usr/libexec/java_home -v 21.0.6) export PATH="$JAVA_HOME/bin:$PATH" ./gradlew :sdks:java:io:google-cloud-platform:test \ --tests org.apache.beam.sdk.io.gcp.pubsub.PubsubIOTest.testBoundedWriteBatchSizeIncludesAttributesAndOrderingKey \ --no-daemon --console=plain ``` ```text BUILD SUCCESSFUL in 33s ``` The captured batch sizes are now `[1, 1]`, so neither publish exceeds the configured limit. The full `PubsubIOTest` class also passes with the same Gradle task and the class-level test selector. - [x] This PR fixes #28011. - [x] `CHANGES.md` is unchanged for this small bug fix. - [x] This is not a large contribution, so the ICLA checklist item does not apply. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
