kennknowles commented on PR #16888:
URL: https://github.com/apache/beam/pull/16888#issuecomment-1103238917
I guess I'd say it the opposite way: Dataflow runner v1 does not support SDF
appropriately.
I don't think it is strange to use SDF in the SDK and override in
DataflowRunner. This is exactly what all runners do: they translate Beam's
primitives to something they can execute. Runners are allowed and encouraged to
directly translate composites as long as they preserve the semantics.
So my summary take is still:
- Dataflow runner v1 doesn't support SDF very well, and I don't really know
the boundaries of where the support actually works vs where it crashes vs where
it fails silently.
- Dataflow runner v1 can still execute Kafka reads by executing the
composite directly in a custom way, and that sort of thing is explicitly a
design goal of Beam.
- If this results in mysterious behavior or inscrutable error messages,
that is a bug.
- If we can positively confirm (via continuously running tests) that
Dataflow runner v1 correctly supports some functionality, then we can enable it.
So I understand the motivation here, but I just don't think it is safe.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
