[
https://issues.apache.org/jira/browse/FLINK-24493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18023929#comment-18023929
]
Rion Williams commented on FLINK-24493:
---------------------------------------
[~fpaul] / [~arvid] / [~dmvk]
I've created a draft pull request for the initial pass for this effort
available below:
* [https://github.com/apache/flink/pull/27072]
There's still quite a bit of work and improvements to make, but I feel pretty
good about this initial pass. The ones that immediately come to mind include:
* Address currently shimmed out implementation for
`DemultiplexingSinkWriter.serializeWriterStates` to ensure that checkpointing
behavior and recovery scenarios are accounted for.
** Extend existing test suite to verify these and potential edge-cases that
may arise related to them.
* Add Test Container-backed integration test suite to verify expected behavior
for popular and widely supported sinks (e.g. Kafka, JDBC, Elasticsearch, etc.)
* Additional documentation and examples
I'd appreciate any and all feedback as it's been a while but since the recent
migration to the Sink API from my initial implementation is fairly fresh in my
mind, I figured it was a better time than any to try and contribute.
Thanks again,
Rion
> Introduce DemultiplexingSink to Support Dynamic Sink Routing
> ------------------------------------------------------------
>
> Key: FLINK-24493
> URL: https://issues.apache.org/jira/browse/FLINK-24493
> Project: Flink
> Issue Type: New Feature
> Components: Connectors / Common
> Reporter: Rion Williams
> Assignee: Rion Williams
> Priority: Major
> Labels: pull-request-available
>
> Recently, FLINK-23977 attempted to introduce an approach to supporting
> dynamic routing for the Elasticsearch sink, however during some discussion
> within [the pull request|https://github.com/apache/flink/pull/17061], the
> idea was introduced to create a more generic approach.
> The idea being that we could introduce a common DemultiplexingSink and
> related interface for handling routing to any number of existing sinks
> similar to the implementation mentioned in FLINK-23977 to the common
> connectors directory.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
