tvalentyn commented on issue #36201: URL: https://github.com/apache/beam/issues/36201#issuecomment-3887773454
> I’m also curious whether a Python equivalent of --experiments=enable_custom_pubsub_sink would be feasible as a shorter-term option Good question, I don't know whether it would only be inefficient or there are some blockers for that. You could ask on Beam mailing list and other folks might have some knowledge about what it would take to make custom pubsub sink to work with Dataflow. I think I would probably recommend such users to use Java's custom pubsub sink in Python SDK via cross-language IO. That sounds like a viable approach and might already work, though I didn't give it a try - feel free to do so. > A few options I see are: I think limiting the change to direct runner is fine; warning users that it won't work on Dataflow is fine; we can use consistent messaging wit Java SDK, as long as we don't introduce breaking changes (e.g. I wouldn't start failing pipelines that are already running even if they attempt to use the functionality that doesn't work yet; a warning might be better). If we go with direct runner approach, let's limit the scope of integration tests to direct runner and also verify they pass. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
