I see where you are coming from. It is truly a marginal feature of Beam at the moment, but really *really* useful in debugging, when a runner takes advantage of it. More inline - FWIW it may seem like I'm defending the tests, but I have been in the same situation and felt the same way. I'm just trying to ground things in specifics.
On Thu, Oct 26, 2017 at 2:24 PM, Eugene Kirpichov < [email protected]> wrote: > Hello, > > DisplayData is currently in an odd position: > - Dataflow is the only runner that supports it at all [1] > - Dataflow supports it only for primitive transforms, but discards it for > composite transforms [2](which is quite problematic for very complex but > commonly used composite transforms, such as TextIO.write() and > BigQueryIO.write()) > As of today, the Java and Python Dataflow runners actually transmit the entire original pipeline, which has slots for display data on composites. So the future in which this is actually used - at least for Dataflow - might be very near. - There are a lot of tests testing display data of composite transforms - > these tests are currently testing something that is a no-op in production; > and these tests often are a maintenance burden when refactoring a > transform. Which composites' tests in particular? Are they legitimate tests? What should we do about this? I see two options: > - Publish guidance for library authors that Beam is not currently ready for > supporting HasDisplayData on composite transforms, and you shouldn't bother > implementing or testing it (though you can implement it on primitive > transforms of course - which is basically just Read.from(Source) and ParDo) > - Say that people should be implementing and testing this feature > nevertheless, and promise that Beam runners are going to make it "worth it" > (not be a no-op) in the foreseeable future. > Option 3: Don't say anyone _should_ do any particular thing, but promise that [some] Beam runners are [probably] going to make it "worth it" pretty soon. I expect third-party transform authors are already not using this, are they? Kenn > [1] > https://lists.apache.org/thread.html/53b33f17d981054ed198af03511969 > 07bf147b6acf16160dbf395b62@%3Cdev.beam.apache.org%3E > [2] https://issues.apache.org/jira/browse/BEAM-366 >
