robertwb commented on code in PR #25935:
URL: https://github.com/apache/beam/pull/25935#discussion_r1145292371
##########
sdks/python/apache_beam/yaml/yaml_transform.py:
##########
@@ -377,6 +378,18 @@ def pipeline_as_composite(spec):
return dict(spec, name=None, type='composite')
+def normalize_source_sink(spec):
Review Comment:
Yeah, the question here is whether source1 would get fed into t1 (with a
flatten), or fed into source2 (e.g. something that takes a PCollection of file
names and reads the elements) or ???. Similarly with sinks (both of which would
behave differently in a composite/pipeline where the inputs are explicitly
named vs. a chain). Also, does this make the "normal/easy" case look odd to
support the "atypical/branched" one?
Or would this be better handled by a new, explicit "allof" transform that
took a list of transforms and implicitly flattened them (which could be placed
in the single source or sink spot)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
