[
https://issues.apache.org/jira/browse/KAFKA-21101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chia-Ping Tsai updated KAFKA-21101:
-----------------------------------
Description: Currently, an invalid record marks the partition
"uncleanable", which blocks the cleaner from compacting all other records in
that partition. Since the record is already invalid regardless of the root
cause, nobody can read it or anything after it in the same batch (consumers
parse a batch sequentially and fail at the same point), so dropping them is a
more targeted fix than deleteRecords, which removes everything before an
offset. We should offer an option to drop the invalid record and the remaining
records of its batch, and keep the partition cleanable. Records before the
invalid one are parsed successfully and are retained as usual. (was:
Currently, an invalid record marks the partition "uncleanable", which blocks
the cleaner from compacting all other records in that partition. Since the
record is already invalid regardless of the root cause, nobody can read it or
anything after it in the same batch (consumers parse a batch sequentially and
fail at the same point), so dropping them loses nothing observable. We should
offer an option to drop the invalid record and the remaining records of its
batch, and keep the partition cleanable. Records before the invalid one are
parsed successfully and are retained as usual.)
> Drop the remaining records of a batch when encountering an invalid record
> -------------------------------------------------------------------------
>
> Key: KAFKA-21101
> URL: https://issues.apache.org/jira/browse/KAFKA-21101
> Project: Kafka
> Issue Type: Improvement
> Reporter: Chia-Ping Tsai
> Assignee: majialong
> Priority: Major
> Labels: need-kip
>
> Currently, an invalid record marks the partition "uncleanable", which blocks
> the cleaner from compacting all other records in that partition. Since the
> record is already invalid regardless of the root cause, nobody can read it or
> anything after it in the same batch (consumers parse a batch sequentially and
> fail at the same point), so dropping them is a more targeted fix than
> deleteRecords, which removes everything before an offset. We should offer an
> option to drop the invalid record and the remaining records of its batch, and
> keep the partition cleanable. Records before the invalid one are parsed
> successfully and are retained as usual.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
