Hi, I notice that there is an experimental lineage integration for Flink with OpenLineage https://openlineage.io/docs/integrations/flink . I think this feature would allow for a superior Flink OpenLineage integration, Kind regards, David.
From: XTransfer <jiabao....@xtransfer.cn.INVALID> Date: Tuesday, 19 September 2023 at 15:47 To: dev@flink.apache.org <dev@flink.apache.org> Subject: [EXTERNAL] Re: [DISCUSS] FLIP-314: Support Customized Job Lineage Listener Thanks Shammon for this proposal. That’s helpful for collecting the lineage of Flink tasks. Looking forward to its implementation. Best, Jiabao > 2023年9月18日 20:56，Leonard Xu <xbjt...@gmail.com> 写道： > > Thanks Shammon for the informations, the comment makes the lifecycle clearer. > +1 > > > Best, > Leonard > > >> On Sep 18, 2023, at 7:54 PM, Shammon FY <zjur...@gmail.com> wrote: >> >> Hi devs, >> >> After discussing with @Qingsheng, I fixed a minor issue of the lineage >> lifecycle in `StreamExecutionEnvironment`. I have added the comment to >> explain that the lineage information in `StreamExecutionEnvironment` will be >> consistent with that of transformations. When users clear the existing >> transformations, the added lineage information will also be deleted. >> >> Please help to review it again, and If there are no more concerns about >> FLIP-314[1], I would like to start voting later, thanks. cc @ <>Leonard >> >> Best, >> Shammon FY >> >> On Mon, Jul 17, 2023 at 3:43 PM Shammon FY <zjur...@gmail.com >> <mailto:zjur...@gmail.com>> wrote: >> Hi devs, >> >> Thanks for all the valuable feedback. If there are no more concerns about >> FLIP-314[1], I would like to start voting later, thanks. >> >> >> [1] >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-314%3A+Support+Customized+Job+Lineage+Listener >> >> <https://cwiki.apache.org/confluence/display/FLINK/FLIP-314%3A+Support+Customized+Job+Lineage+Listener >> > >> >> Best, >> Shammon FY >> >> >> On Wed, Jul 12, 2023 at 11:18 AM Shammon FY <zjur...@gmail.com >> <mailto:zjur...@gmail.com>> wrote: >> Thanks for the valuable feedback, Leonard. >> >> I have discussed with Leonard off-line. We have reached some conclusions >> about these issues and I have updated the FLIP as follows: >> >> 1. Simplify the `LineageEdge` interface by creating an edge from one source >> vertex to sink vertex. >> 2. Remove the `TableColumnSourceLineageVertex` interface and update >> `TableColumnLineageEdge` to create an edge from columns in one source to >> each sink column. >> 3. Rename `SupportsLineageVertex` to `LineageVertexProvider` >> 4. Add method `addLineageEdges(LineageEdge ... edges)` in >> `StreamExecutionEnviroment` for datastream job and remove previous methods >> in `DataStreamSource` and `DataStreamSink`. >> >> Looking forward to your feedback, thanks. >> >> Best, >> Shammon FY > Unless otherwise stated above: IBM United Kingdom Limited Registered in England and Wales with number 741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
