The need for composite PValues is closely tied to the language of the SDK, but not an intrinsic part of the to-be-executed pipeline (e.g. Python doesn't have PCollectionList because a list of PCollections does the job just fine (better, in fact), but Java needs it to preserve static typing).
So +1 on making the pipeline representation should be as simple as possible (but no simpler :). On Tue, Jan 17, 2017 at 11:49 AM, Amit Sela <[email protected]> wrote: > This looks pretty straight-forward, let me know if you need anything in the > Spark runner front here. > +1 > > On Tue, Jan 17, 2017 at 9:13 PM Kenneth Knowles <[email protected]> > wrote: > >> This is a nice concrete (and inevitable) step towards our SDK-agnostic >> pipeline representation. >> >> +1 from me! >> >> On Tue, Jan 17, 2017 at 10:46 AM, Thomas Groh <[email protected]> >> wrote: >> >> > Hey everyone; >> > >> > I've been working on parts of the runner API recently, and part of that >> has >> > included a shift of how composite inputs and outputs must be represented >> by >> > the time a PipelineRunner begins to access them. I have a PR that >> completes >> > this work within the Java SDK, but wanted to ensure that everyone agrees >> on >> > the change and anything required on their end before I start fiddling >> with >> > all of the runner internals. For anyone except current runner authors, >> this >> > should be completely transparent; for current runner authors, I need a >> > short code review but nothing else. >> > >> > I've written a one-pager about what's changing; the link is at >> > https://s.apache.org/beam-runner-composites >> > >> > or directly at >> > https://docs.google.com/document/d/1_CHLnj1RFAGKy_ >> > MfR54XmixakYNmCnhGZLWmuDSMJ10/edit#heading=h.qlkikisrzqqf >> > >> > Thanks, >> > >> > Thomas >> > >>
