[
https://issues.apache.org/jira/browse/KAFKA-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17042181#comment-17042181
]
Sophie Blee-Goldman commented on KAFKA-8037:
--------------------------------------------
That's a good thought actually – I'm not sure the length would be that
prohibitive even, since we would only need to store the bad offsets since the
last commit in each commit metadata. Even if most of them are bad, and we
process a relatively large number of records between each commit, we could
encode the bad offsets pretty compactly within the restricted range of offsets
(ie, we don't have to encode the full value for each offset). And at most we'd
have to encode only half, since we could swap to encoding only the "good"
offsets if most of them are bad. We could definitely get even more clever and
encode them quite densely, but I think the simple approach may be sufficient.
> KTable restore may load bad data
> --------------------------------
>
> Key: KAFKA-8037
> URL: https://issues.apache.org/jira/browse/KAFKA-8037
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Reporter: Matthias J. Sax
> Priority: Minor
> Labels: pull-request-available
>
> If an input topic contains bad data, users can specify a
> `deserialization.exception.handler` to drop corrupted records on read.
> However, this mechanism may be by-passed on restore. Assume a
> `builder.table()` call reads and drops a corrupted record. If the table state
> is lost and restored from the changelog topic, the corrupted record may be
> copied into the store, because on restore plain bytes are copied.
> If the KTable is used in a join, an internal `store.get()` call to lookup the
> record would fail with a deserialization exception if the value part cannot
> be deserialized.
> GlobalKTables are affected, too (cf. KAFKA-7663 that may allow a fix for
> GlobalKTable case). It's unclear to me atm, how this issue could be addressed
> for KTables though.
> Note, that user state stores are not affected, because they always have a
> dedicated changelog topic (and don't reuse an input topic) and thus the
> corrupted record would not be written into the changelog.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
