[
https://issues.apache.org/jira/browse/KAFKA-10179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17140328#comment-17140328
]
Bruno Cadonna commented on KAFKA-10179:
---------------------------------------
[~desai.p.rohan] While I find the idea of optimizing the materialization in the
deserializer intriguing, I think the performance penalty that we would pay by
deserializing and serializing each record during restoration is not worthwhile.
Additionally -- if optimization is turned on -- we would need to read the
original data from the source topic instead of the projected data from the
changelog topic during each restoration which would again hit performance. Of
course, we would need experiments to better understand the implications.
An alternative idea would be to allow to plugin a byte-based transformation
that does not need to deserialize and serialize each record. However, that
would not solve the issue of having to read the unprojected data during each
restoration.
If you are concerned with the amount of data to materialize a solution could be
to optimize on topology-level by introducing a {{map()}} that makes the
projection followed by a {{toTable()}} to materialize the data. That data read
from the input topic would be the unprojected data but the one materialized is
the projected one and also during restoration we would just read the projected
data. An additional advantage of this method is that you can leave the source
table optimization turned on, because it would not apply to this case.
In summary, the source table optimization was not introduced for the case you
describe. IMO, it is not even an optimization in that case.
> State Store Passes Wrong Changelog Topic to Serde for Optimized Source Tables
> -----------------------------------------------------------------------------
>
> Key: KAFKA-10179
> URL: https://issues.apache.org/jira/browse/KAFKA-10179
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 2.5.0
> Reporter: Bruno Cadonna
> Assignee: Bruno Cadonna
> Priority: Major
> Fix For: 2.7.0
>
>
> {{MeteredKeyValueStore}} passes the name of the changelog topic of the state
> store to the state store serdes. Currently, it always passes {{<application
> ID>-<store name>-changelog}} as the changelog topic name. However, for
> optimized source tables the changelog topic is the source topic.
> Most serdes do not use the topic name passed to them. However, if the serdes
> actually use the topic name for (de)serialization, e.g., when Kafka Streams
> is used with Confluent's Schema Registry, a
> {{org.apache.kafka.common.errors.SerializationException}} is thrown.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
