[
https://issues.apache.org/jira/browse/BEAM-7883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16911473#comment-16911473
]
Yurii Atamanchuk commented on BEAM-7883:
----------------------------------------
Thanks!
Yeah, another option can be to add something like `public abstract byte[]
encodePayload(byte[] payload)` into PubsubClient to get better estimation (esp
for Json clinet) of message size, but probably it's not really worth it :)
> PubsubIO (Java) write batch size can exceed request payload limit
> -----------------------------------------------------------------
>
> Key: BEAM-7883
> URL: https://issues.apache.org/jira/browse/BEAM-7883
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp
> Affects Versions: 2.13.0
> Reporter: Yurii Atamanchuk
> Priority: Minor
>
> In some (probably rare) cases PubsubIO write (in Batch mode) batch size can
> exceed request payload limit of 10mb. PubsubIO ensures that batch size is
> less than limit (10mb by default). But then PubsubJsonClient is used that
> converts message payloads into URL-Safe Base64 encoding which can inflate
> message size (in my case for json strings it was up to 25-30%). As result we
> get 400 response (with 'Request payload size exceeds the limit: 10485760
> bytes' message), even though original batch had correct size.
> Obvious workaround is to reduce batch size
> (`PubsubIO.writeMessages().to(...).withMaxBatchBytesSize(... i.e. 5mb ...)`),
> but it is a bit annoying.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
