scwhittle commented on code in PR #36027:
URL: https://github.com/apache/beam/pull/36027#discussion_r2332527387
##########
sdks/python/apache_beam/io/gcp/pubsub.py:
##########
@@ -541,11 +546,75 @@ def is_bounded(self):
return False
-# TODO(BEAM-27443): Remove in favor of a proper WriteToPubSub transform.
+class _PubSubWriteDoFn(DoFn):
+ """DoFn for writing messages to Cloud Pub/Sub.
+
+ This DoFn handles both streaming and batch modes by buffering messages
+ and publishing them in batches to optimize performance.
+ """
+ BUFFER_SIZE_ELEMENTS = 100
+ FLUSH_TIMEOUT_SECS = BUFFER_SIZE_ELEMENTS * 0.5
Review Comment:
seems odd to base this on the buffer size, just put a value
also since this is just a safeguard against stuckness I would set it to
something like 5 minutes so jobs don't fail unnecessarily
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
