[ https://issues.apache.org/jira/browse/BEAM-2660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kenneth Knowles updated BEAM-2660: ---------------------------------- Fix Version/s: 2.9.0 > Set PubsubIO batch size using builder > ------------------------------------- > > Key: BEAM-2660 > URL: https://issues.apache.org/jira/browse/BEAM-2660 > Project: Beam > Issue Type: Improvement > Components: io-java-gcp > Reporter: Carl McGraw > Assignee: Chamikara Jayalath > Priority: Major > Labels: gcp, java, pubsub, sdk > Fix For: 2.9.0 > > Time Spent: 4h > Remaining Estimate: 0h > > PubsubIO doesn't allow users to set the publish batch size. Instead the value > is hard coded in both the BoundedPubsubWriter and the UnboundedPubsubSink. > google's pub/sub is bound to a maximum of 10mb per request size. My company > has run into problems with events that are individually smaller than 1mb, but > when batched in the 100 or 2000 default batch sizes causes pubsub to fail to > send the event. -- This message was sent by Atlassian JIRA (v7.6.3#76005)