Hello Beam users, I’m wondering if there’s a way to reliably rate-limit publish requests to a Pub/Sub topic in Apache Beam (I am using Dataflow runner).
My pipeline looks roughly like this: Generate messages every 5 minutes → rate limit to 100 messages/sec → publish to Pub/Sub Because messages are generated in bursts every 5 minutes, I’m trying to smooth out the publish rate. I’ve applied a rate-limiting step and I’m using PubsubIO.writeMessages().withMaxBatchSize(100) to limit batch size. However, I still observe in Pub/sub topic metrics that: - The publish rate exceeds 100 messages/sec - The average number of messages per batch is also greater than 100 Does anyone have ideas on how this could happen, or suggestions for better controlling the publish rate in this scenario? Thanks for your help, Shaochen -- --- This email is confidential/privileged. If you're not the intended recipient, please delete it and notify us immediately; please do not copy/use/disclose it for any purpose, to anyone. Thank you!
