[
https://issues.apache.org/jira/browse/FLINK-39813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-39813:
-----------------------------------
Labels: pull-request-available (was: )
> [Connectors/Kinesis] Add lineage support to Kinesis Streams connector
> ---------------------------------------------------------------------
>
> Key: FLINK-39813
> URL: https://issues.apache.org/jira/browse/FLINK-39813
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Kinesis
> Affects Versions: aws-connector-6.0.0
> Reporter: Francisco Morillo
> Priority: Major
> Labels: pull-request-available
>
> The Kinesis Streams connector (flink-connector-aws-kinesis-streams) does
> not implement the LineageVertexProvider
> interface introduced in Flink 2.0. This means lineage tools like
> OpenLineage cannot automatically extract dataset
> information from Flink jobs using KDS sources and sinks.
> The Kafka connector already implements this successfully. This change adds
> the same capability to the Kinesis Streams
> connector.
> Implementation:
> - KinesisStreamsSource implements LineageVertexProvider, returning
> SourceLineageVertex with namespace derived from
> stream ARN
> - KinesisStreamsSink implements LineageVertexProvider, returning
> LineageVertex with the same ARN-based namespace
> - New utility package: KinesisLineageUtil, KinesisDatasetFacet,
> TypeDatasetFacet
> - Unit tests covering source, sink, and utility methods
> Namespace format: arn:\{partition}:kinesis:\{region}:\{account}:stream
> Dataset name: stream name extracted from the ARN
> This covers:
> - DataStream API (KinesisStreamsSource/Sink implement the interface
> directly)
> - SQL/Table API (KinesisDynamicSource internally creates
> KinesisStreamsSource, planner extracts lineage automatically)
> End-to-end tested with OpenLineage flink2 module (v1.47.1) confirming
> lineage events are emitted with correct
> namespace and stream name when using the DataStream API.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
