twang126 commented on code in PR #25935:
URL: https://github.com/apache/beam/pull/25935#discussion_r1145368828
##########
sdks/python/apache_beam/yaml/yaml_transform.py:
##########
@@ -377,6 +378,18 @@ def pipeline_as_composite(spec):
return dict(spec, name=None, type='composite')
+def normalize_source_sink(spec):
Review Comment:
I see, now I realize I had a misunderstanding of chain. I had thought
earlier that the goal was to make chain support everything, but now see
there'll be a whole other family of composite transforms that can handle non
linear graphs. Side question - can a single pipeline combine a chain with a
different composite transform? i.e. define multiple composite transforms that
then feeds into a chain all within a single pipeline? From what I've
interpreted I think the answer is yes?
I thought about it again and I really like the idea to have one source and
sink spot **and** to have that allof transform.
We get a really easy simple path (i.e. linear graph, 1 sink and source), and
the allof transform gives the YAML dsl enough expressiveness to expand that to
multiple sources & sinks in a clean way. This would already cover basically all
of the expressiveness we had at Lyft. I'm also not worried about the implicit
flatten in the allof transform because imo forcing the user to use allof and
documenting what it does would make that flatten explicit.
Which way are you leaning?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
