bvolpato opened a new issue, #23022: URL: https://github.com/apache/beam/issues/23022
### What happened? There is currently a limit of 10 MB for the `data` field on Pub/Sub, as well as a limit for the entire request, which is also 10 MB [1]. If attributes are being used, `PubsubIO` does not take them into account and can go over the limit, leading to some exceptions: ``` Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 400 Bad Request POST https://pubsub.googleapis.com/v1/projects/{project_name}/topics/{topic_name}:publish { "code" : 400, "errors" : [ { "domain" : "global", "message" : "Request payload size exceeds the limit: 10485760 bytes.", "reason" : "badRequest" } ], "message" : "Request payload size exceeds the limit: 10485760 bytes.", "status" : "INVALID_ARGUMENT" ``` Also related to that, it is important to note that limits for attribute key and values are in bytes, and the current comparison accounts for the string length. So even though there are currently checks in place [2], they are allowing some attributes that use non-ASCII characters to not be counted as more than one byte: ``` "message" : "The attribute \"description\" in the request has a value that is too long. The length is 1028 characters, but the maximum allowed is 1024. Refer to https://cloud.google.com/pubsub/quotas for more information.", "status" : "INVALID_ARGUMENT" ``` [1] https://cloud.google.com/pubsub/quotas#resource_limits [2] https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.java#L202-L218 ### Issue Priority Priority: 1 ### Issue Component Component: io-java-gcp -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
