[
https://issues.apache.org/jira/browse/KAFKA-9260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16990279#comment-16990279
]
Matthias J. Sax commented on KAFKA-9260:
----------------------------------------
That sounds like a good fix! However, I don't think we can remove any
`KafkaStreams` constructor, only the parameter-less `StreamsBuilder#build()`
method, because if one uses the PAPI, she still needs to be able to pass in the
config into `KafkaStreams`.
The only alternative might be, to require the constructor of `Topology` to take
the corresponding properties but it seems more clumsy than helpful to me. But
if we follow this path, we need a KIP anyway and can discuss the details there.
> Improve Serde "push down" and "wrapping"
> ----------------------------------------
>
> Key: KAFKA-9260
> URL: https://issues.apache.org/jira/browse/KAFKA-9260
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Reporter: Matthias J. Sax
> Priority: Major
>
> Kafka Streams DSL supports "serde push down" feature to let downstream
> operators inherit upstream serdes if key and/or value are not modified.
> Furthermore, some operators "wrap" user specified serdes internally (eg,
> windowed aggregations wrap the user specified key-serde with a time/session
> window serde – some stores use `ValueAndTimestampSerde` and foreign-key joins
> also uses some internal wrappers).
> The current implementation faces couple of issues, because the "serde push
> down" feature is a DSL level feature that is used when the Topology is
> generated. Furthermore, "serde wrapping" is an operator specific feature, not
> a DSL concept per-se. At runtime, neither "push down" nor "wrapping" are know
> concepts.
> This design implies that if users specify serdes, wrapping and push down
> works as expected. However, if we fall back to default serdes, special care
> needs to be taken: for example, some operators not apply the wrapping logic
> during translation time, and there is additional code that does the wrapping
> of default serdes as runtime. Another approach would be to wrap a null-Serde,
> and update the wrapper later (ie, overwrite `null` with the default serde
> from the config).
> Overall, the current design leads to bugs (eg, KAFKA-9248 and KAFKA-9259),
> and user confusion how it actually works and when/where to specify serdes.
> Hence, we should consider to rework how we do serde push down and/or wrapping.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
