charlespnh commented on issue #33981: URL: https://github.com/apache/beam/issues/33981#issuecomment-3530712760
> Metric storage seems to be a "third wheel" when the goal is to submit lineage data to OpenLineage compliant API To submit lineage data in OpenLineage format, you need to know the source-sink pairs. During pipeline graph traversal you need to know which PTransforms are IOs that report the lineage data. Lineage, being emitted as metric, is scoped by the transform that emits it so you know which transforms are IOs that report lineage data while traversing the graph. In your plug-in approach, to my knowledge the lineage will similarly need to be scoped to its transform when being stored, so you'll probably end up reinventing the wheel by implementing some aspect of the metric storage in Beam today... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
