Kevin Lam created FLINK-35808:
---------------------------------
Summary: Let ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG be
overridable by user in KafkaSourceBuilder
Key: FLINK-35808
URL: https://issues.apache.org/jira/browse/FLINK-35808
Project: Flink
Issue Type: Improvement
Components: Connectors / Kafka
Affects Versions: kafka-3.2.0
Reporter: Kevin Lam
This issue is a follow-up to [this mailing list
discussion|https://lists.apache.org/thread/spl88o63sjm2dv4l5no0ym632d2yt2o6].
I'd like to propose letting the ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG
be overridable by user in KafkaSourceBuilder, as shown in this DRAFT PR:
https://github.com/apache/flink-connector-kafka/pull/108
>From the PR description:
{quote}This allows users to easily implement the [{{claim check}} large message
pattern|https://developer.confluent.io/patterns/event-processing/claim-check/]
without bringing any concerns into the Flink codebase otherwise, by specifying
a {{value.deserializer}} that handles it, but otherwise passes through the
bytes.
Note: [overriding {{value.serializer}} is already supported on the Producer
side.
|https://github.com/apache/flink-connector-kafka/blob/15d3fbd4e65dae6c334e2386dd337d2bf423c216/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/sink/KafkaSinkBuilder.java#L82-L83]Other
Reading:
[https://www.kai-waehner.de/blog/2020/08/07/apache-kafka-handling-large-messages-and-files-for-image-video-audio-processing/
]
[https://www.conduktor.io/kafka/how-to-send-large-messages-in-apache-kafka/#Option-1:-using-an-external-store-(GB-size-messages)-0]{quote}
What do folks think? If it seems reasonable I can follow the steps in the
[contribution guide|https://flink.apache.org/how-to-contribute/overview/].
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
