Kevin Tseng created FLINK-33479:
-----------------------------------
Summary: Add Rerouting support to KafkaSink
Key: FLINK-33479
URL: https://issues.apache.org/jira/browse/FLINK-33479
Project: Flink
Issue Type: Improvement
Components: Connectors / Kafka
Affects Versions: 1.18.0
Reporter: Kevin Tseng
KafkaSink currently operates in a fail / success model. If it fails it will
throw FlinkRuntimeException, any record that may cause failure has to be
filtered by the operators preceeding KafkaSink. If there's unexpected failure
that was caught, this will create continuous failure until a hotfix has been
provided and issue addressed.
Relying on checkpoint to restore the process also create duplicated records
produced within the interval, unless EXACTLY_ONCE is utilized.
Since Flink is meant to be a real-time streaming process, any downtime can
create detrimental effect to the downstream consumer of the data.
I'm proposing to add an error handling injection/intercept to KafkaSink (user
can define an error handler or route the error causing record to another
downstream operator) to keep the process running. Only throwing
RuntimeException when the interceptor was not provided.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
