[
https://issues.apache.org/jira/browse/FLINK-31275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18119024#comment-18119024
]
jiawei xu commented on FLINK-31275:
-----------------------------------
Hi Flink maintainers,
I would like to discuss a possible follow-up to FLIP-314 and FLINK-31275.
FLIP-314 introduced the customized job lineage listener and the initial
table-level lineage model. However, for Flink SQL and Table API jobs, there is
currently no generally available and reliable native column lineage result that
users can consume through the standard lineage listener path.
In particular, users cannot consistently obtain:
- which input columns contribute to each output column;
- direct and indirect column dependencies;
- column dependencies introduced by filters and expressions;
- column lineage for StatementSet sink branches;
- column lineage after restoring a compiled plan;
- the same column lineage information when the job is submitted to a remote
Session Cluster.
This makes table-level lineage insufficient for impact analysis, column-level
data governance, sensitive-field tracking, and downstream dependency analysis.
Based on this gap, I implemented a prototype that extends the existing
Flink lineage path with Planner-derived column lineage.
The work includes:
- extracting column relations from the Flink Planner;
- representing direct and indirect column dependencies;
- preserving filter and expression transformations;
- keeping column lineage independent for multiple StatementSet sinks;
- preserving lineage across compiled-plan restore;
- carrying the Planner lineage graph through the remote submission path;
- preserving connector-safe table metadata needed by downstream lineage
consumers;
- keeping lineage processing isolated from Flink execution.
The implementation is available here:
Flink:
https://github.com/Xuxiaotuan/flink/tree/codex/flink-openlineage-column-lineage
OpenLineage:
https://github.com/Xuxiaotuan/OpenLineage/tree/codex/flink-openlineage-column-lineage
I reviewed PR #28002 / FLINK-39491 because it is related to the
Dispatcher-side lineage listener delivery path. However, my primary scope is
the extension of the existing FLIP-314 table-level lineage model with native
column lineage. The Dispatcher transport work is an enabling implementation
detail for making the Planner-generated column lineage available to remote
jobs.
Before preparing a detailed design proposal, could you please advise:
1. Is native Planner-derived column lineage a suitable follow-up scope for
FLIP-314?
2. Should this work continue under FLINK-31275, or should a new JIRA be
created?
3. Should column lineage be specified as an extension of the existing lineage
interfaces?
4. Should the design discussion be added as a follow-up to FLIP-314, or
prepared as a separate FLIP?
5. Should remote Session Cluster delivery be included in the same proposal, or
handled as a separate implementation concern?
I would appreciate guidance on the appropriate issue and proposal structure
before preparing the detailed design.
> Flink supports reporting and storage of source/sink tables relationship
> -----------------------------------------------------------------------
>
> Key: FLINK-31275
> URL: https://issues.apache.org/jira/browse/FLINK-31275
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Planner
> Affects Versions: 1.18.0
> Reporter: Fang Yong
> Assignee: Fang Yong
> Priority: Major
> Labels: pull-request-available
>
> FLIP-314 has been accepted
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-314%3A+Support+Customized+Job+Lineage+Listener
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
