[
https://issues.apache.org/jira/browse/FLINK-31275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17807486#comment-17807486
]
Zhenqiu Huang commented on FLINK-31275:
---------------------------------------
Hi Everyone, I also want to bump the thread due to the internal needs. I feel
open lineage community gives very good suggestion to define an intermediate
representation (Dataset) about the metadata of a since/sink. Also LineageVertex
could definitely have multiple dataset, for example Hybrid source users who
read from Kafka first then switch to iceberg. Given this, I feel the config
should be in dataset rather than LineageVertex. On the other hand, we want to
make the column lineage possible, so having the query in the dataset will be
reason for lineage provide to analysis the column relationship. For
input/output schema, we may put it into a facet. It could be optional depends
on the connector implementation. How do you think [~zjureel] [~mobuchowski]?
public interface LineageVertex {
/* List of input (for source) or output (for sink) datasets interacted with
by the connector */
List<Dataset> datasets;
}
public interface Dataset {
/* Name for this particular dataset. */
String name;
/* Unique name for this dataset's datasource. */
String namespace;
/* Query used to generate the dataset If there is */
String query;
/* Facets for the lineage vertex to describe the particular information of
dataset. */
Map<FacetType, Facet> facets;
}
Facet type could be SchemaFacet and ConfigFacet.
> Flink supports reporting and storage of source/sink tables relationship
> -----------------------------------------------------------------------
>
> Key: FLINK-31275
> URL: https://issues.apache.org/jira/browse/FLINK-31275
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Planner
> Affects Versions: 1.18.0
> Reporter: Fang Yong
> Assignee: Fang Yong
> Priority: Major
>
> FLIP-314 has been accepted
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-314%3A+Support+Customized+Job+Lineage+Listener
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
