> > As of today, the Java and Python Dataflow runners actually transmit the > entire original pipeline, which has slots for display data on composites. > So the future in which this is actually used - at least for Dataflow - > might be very near. >
I am not 100% sure, but I believe this is not true. The workflow graph does not contain composites. It only contains primitives, and dataflow uses their names to infer composites (e.g. 'Do/DoFirst' and 'Do/DoSecond' are inferred to belong to the same composite because they share their first level translform). Can someone confirm whether this is the case? If so, the workflow graph representation for Dataflow would need to change to explicitly include display data for composites. Nonetheless, display data is very useful, so I'd think it's useful to make it available in other runners, and for composites.. Best -P. > > > - There are a lot of tests testing display data of composite transforms - > > these tests are currently testing something that is a no-op in > production; > > and these tests often are a maintenance burden when refactoring a > > transform. > > > Which composites' tests in particular? Are they legitimate tests? > > > What should we do about this? I see two options: > > - Publish guidance for library authors that Beam is not currently ready > for > > supporting HasDisplayData on composite transforms, and you shouldn't > bother > > implementing or testing it (though you can implement it on primitive > > transforms of course - which is basically just Read.from(Source) and > ParDo) > > - Say that people should be implementing and testing this feature > > nevertheless, and promise that Beam runners are going to make it "worth > it" > > (not be a no-op) in the foreseeable future. > > > > Option 3: Don't say anyone _should_ do any particular thing, but promise > that [some] Beam runners are [probably] going to make it "worth it" pretty > soon. I expect third-party transform authors are already not using this, > are they? > > Kenn > > > > > [1] > > https://lists.apache.org/thread.html/53b33f17d981054ed198af03511969 > > 07bf147b6acf16160dbf395b62@%3Cdev.beam.apache.org%3E > > [2] https://issues.apache.org/jira/browse/BEAM-366 > > >
