[
https://issues.apache.org/jira/browse/FLINK-24905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zichen Liu updated FLINK-24905:
-------------------------------
Description:
h2. Motivation
*User stories:*
As a Flink user, I’d like to use the Table API for the new Kinesis Data Streams
sink.
*Scope:*
* Introduce {{AsyncDynamicTableSink}} that enables Sinking Tables into Async
Implementations.
* Implement a new {{KinesisDynamicTableSink}} that uses
{{KinesisDataStreamSink}} Async Implementation and implements
{{{}AsyncDynamicTableSink{}}}.
* The implementation introduces Async Sink configurations as optional options
in the table definition, with default values derived from the
{{KinesisDataStream}} default values.
* Unit/Integration testing. modify KinesisTableAPI tests for the new
implementation, add unit tests for {{AsyncDynamicTableSink}} and
{{KinesisDynamicTableSink}} and {{{}KinesisDynamicTableSinkFactory{}}}.
* Java / code-level docs.
h2. References
More details to be found
[https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink]
*Update:*
^^ Status Update ^^
__List of all work outstanding for 1.15 release__
[Merged] https://github.com/apache/flink/pull/18165 - KDS DataStream Docs
[Merged] https://github.com/apache/flink/pull/18396 - [hotfix] for infinte loop
if not flushing during commit
[Merged] https://github.com/apache/flink/pull/18421 - Mark Kinesis Producer as
deprecated (Prod: FLINK-24227)
[Merged] https://github.com/apache/flink/pull/18348 - KDS Table API Sink & Docs
[Merged] https://github.com/apache/flink/pull/18488 - base sink retry entries
in order not in reverse
[Merged] https://github.com/apache/flink/pull/18512 - changing failed requests
handler to accept List in AsyncSinkWriter
[Merged] https://github.com/apache/flink/pull/18483 - Do not expose the element
converter
[Merged] https://github.com/apache/flink/pull/18468 - Adding Kinesis data
streams sql uber-jar
Ready for review:
[SUCCESS ] https://github.com/apache/flink/pull/18314 - KDF DataStream Sink &
Docs
[BLOCKED on ^^ ] https://github.com/apache/flink/pull/18426 - rename
flink-connector-aws-kinesis-data-* to flink-connector-aws-kinesis-* (module
names) and KinesisData*Sink to Kinesis*Sink (class names)
Pending PR:
* Firehose Table API Sink & Docs
* KDF Table API SQL jar
TBD:
* FLINK-25846: Not shutting down
* FLINK-25848: Validation during start up
* FLINK-25792: flush() bug
* FLINK-25793: throughput exceeded
* Update the defaults of KDS sink and update the docs + do the same for KDF
* add a `AsyncSinkCommonConfig` class (to wrap the 6 params) to the
`flink-connector-base` and propagate it to the two connectors
------------- feature freeze
* KDS performance testing
* KDF performance testing
* Clone the new docs to .../contents.zh/... and add the location to the
corresponding Chinese translation jira - KDS -
* Rename [Amazon AWS Kinesis Streams] to [Amazon Kinesis Data Streams] in docs
(legacy issue)
------------- Flink 1.15 release
* KDS end to end sanity test - hits aws apis rather than local docker images
* KDS Python wrappers
* FLINK-25733 - Create A migration guide for Kinesis Table API connector - can
happen after 1.15
* If `endpoint` is provided, `region` should not be required like it currently
is
* Test if Localstack container requires the 10000ms timeout
* Adaptive level of logging (in discussion)
FYI:
* FLINK-25661 - Add Custom Fatal Exception handler in AsyncSinkWriter -
https://github.com/apache/flink/pull/18449
* https://issues.apache.org/jira/browse/FLINK-24229 DDB Sink
Chinese translation:
https://issues.apache.org/jira/browse/FLINK-25735 - KDS DataStream Sink
https://issues.apache.org/jira/browse/FLINK-25736 - KDS Table API Sink
https://issues.apache.org/jira/browse/FLINK-25737 - KDF DataStream Sink
was:
h2. Motivation
*User stories:*
As a Flink user, I’d like to use the Table API for the new Kinesis Data Streams
sink.
*Scope:*
* Introduce {{AsyncDynamicTableSink}} that enables Sinking Tables into Async
Implementations.
* Implement a new {{KinesisDynamicTableSink}} that uses
{{KinesisDataStreamSink}} Async Implementation and implements
{{{}AsyncDynamicTableSink{}}}.
* The implementation introduces Async Sink configurations as optional options
in the table definition, with default values derived from the
{{KinesisDataStream}} default values.
* Unit/Integration testing. modify KinesisTableAPI tests for the new
implementation, add unit tests for {{AsyncDynamicTableSink}} and
{{KinesisDynamicTableSink}} and {{{}KinesisDynamicTableSinkFactory{}}}.
* Java / code-level docs.
h2. References
More details to be found
[https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink]
> [FLIP-171] KDS implementation of Async Sink Table API
> -----------------------------------------------------
>
> Key: FLINK-24905
> URL: https://issues.apache.org/jira/browse/FLINK-24905
> Project: Flink
> Issue Type: New Feature
> Components: Connectors / Kinesis
> Reporter: Zichen Liu
> Assignee: Ahmed Hamdy
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.15.0
>
>
> h2. Motivation
> *User stories:*
> As a Flink user, I’d like to use the Table API for the new Kinesis Data
> Streams sink.
> *Scope:*
> * Introduce {{AsyncDynamicTableSink}} that enables Sinking Tables into Async
> Implementations.
> * Implement a new {{KinesisDynamicTableSink}} that uses
> {{KinesisDataStreamSink}} Async Implementation and implements
> {{{}AsyncDynamicTableSink{}}}.
> * The implementation introduces Async Sink configurations as optional
> options in the table definition, with default values derived from the
> {{KinesisDataStream}} default values.
> * Unit/Integration testing. modify KinesisTableAPI tests for the new
> implementation, add unit tests for {{AsyncDynamicTableSink}} and
> {{KinesisDynamicTableSink}} and {{{}KinesisDynamicTableSinkFactory{}}}.
> * Java / code-level docs.
> h2. References
> More details to be found
> [https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink]
>
> *Update:*
> ^^ Status Update ^^
> __List of all work outstanding for 1.15 release__
> [Merged] https://github.com/apache/flink/pull/18165 - KDS DataStream Docs
> [Merged] https://github.com/apache/flink/pull/18396 - [hotfix] for infinte
> loop if not flushing during commit
> [Merged] https://github.com/apache/flink/pull/18421 - Mark Kinesis Producer
> as deprecated (Prod: FLINK-24227)
> [Merged] https://github.com/apache/flink/pull/18348 - KDS Table API Sink &
> Docs
> [Merged] https://github.com/apache/flink/pull/18488 - base sink retry entries
> in order not in reverse
> [Merged] https://github.com/apache/flink/pull/18512 - changing failed
> requests handler to accept List in AsyncSinkWriter
> [Merged] https://github.com/apache/flink/pull/18483 - Do not expose the
> element converter
> [Merged] https://github.com/apache/flink/pull/18468 - Adding Kinesis data
> streams sql uber-jar
> Ready for review:
> [SUCCESS ] https://github.com/apache/flink/pull/18314 - KDF DataStream Sink &
> Docs
> [BLOCKED on ^^ ] https://github.com/apache/flink/pull/18426 - rename
> flink-connector-aws-kinesis-data-* to flink-connector-aws-kinesis-* (module
> names) and KinesisData*Sink to Kinesis*Sink (class names)
> Pending PR:
> * Firehose Table API Sink & Docs
> * KDF Table API SQL jar
> TBD:
> * FLINK-25846: Not shutting down
> * FLINK-25848: Validation during start up
> * FLINK-25792: flush() bug
> * FLINK-25793: throughput exceeded
> * Update the defaults of KDS sink and update the docs + do the same for KDF
> * add a `AsyncSinkCommonConfig` class (to wrap the 6 params) to the
> `flink-connector-base` and propagate it to the two connectors
> ------------- feature freeze
> * KDS performance testing
> * KDF performance testing
> * Clone the new docs to .../contents.zh/... and add the location to the
> corresponding Chinese translation jira - KDS -
> * Rename [Amazon AWS Kinesis Streams] to [Amazon Kinesis Data Streams] in
> docs (legacy issue)
> ------------- Flink 1.15 release
> * KDS end to end sanity test - hits aws apis rather than local docker images
> * KDS Python wrappers
> * FLINK-25733 - Create A migration guide for Kinesis Table API connector -
> can happen after 1.15
> * If `endpoint` is provided, `region` should not be required like it
> currently is
> * Test if Localstack container requires the 10000ms timeout
> * Adaptive level of logging (in discussion)
> FYI:
> * FLINK-25661 - Add Custom Fatal Exception handler in AsyncSinkWriter -
> https://github.com/apache/flink/pull/18449
> * https://issues.apache.org/jira/browse/FLINK-24229 DDB Sink
> Chinese translation:
> https://issues.apache.org/jira/browse/FLINK-25735 - KDS DataStream Sink
> https://issues.apache.org/jira/browse/FLINK-25736 - KDS Table API Sink
> https://issues.apache.org/jira/browse/FLINK-25737 - KDF DataStream Sink
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
