[
https://issues.apache.org/jira/browse/KAFKA-20845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Jacot resolved KAFKA-20845.
---------------------------------
Fix Version/s: 4.3.2
Resolution: Fixed
> Consumer group downgrades can leave group in invalid state when classic group
> metadata is very large
> ----------------------------------------------------------------------------------------------------
>
> Key: KAFKA-20845
> URL: https://issues.apache.org/jira/browse/KAFKA-20845
> Project: Kafka
> Issue Type: Bug
> Components: group-coordinator
> Affects Versions: 4.2.0, 4.3.0, 4.0.2, 4.1.1, 4.4.0
> Reporter: Sean Quah
> Assignee: Sean Quah
> Priority: Blocker
> Fix For: 4.4.0, 4.3.2
>
>
> Introduced by KAFKA-19760 while attempting to fix writes for large
> compressible records.
> When downgrading a consumer group, we apply changes to the group coordinator
> state directly and do not replay the produced records. However, when
> appending a large but compressible batch, we can allocate an empty batch,
> then flush, then allocate another empty batch. Flushing an empty batch
> silently fails the empty batch and rolls back coordinator state. We then
> write the records for the downgrade, which leaves the in-memory state
> inconsistent with the state on disk.
> After further writes, the state on disk becomes unloadable because the next
> consumer group records cannot be applied to the classic group.
> # We should not flush an empty batch when appending large batches.
> # We could consider succeeding empty batches instead.
> NB: Both succeeding and failing empty batches leads to divergence between
> in-memory state and disk state when there are write operations that update
> the state directly without replay.
> When succeeding empty batches, a write operation that updates state directly
> and then fails to serialize its records could leave an empty batch. When the
> batch is committed, we end up with in-memory changes without corresponding
> records on disk.
> When failing empty batches, a lingering empty batch followed by a write
> operation that updates state directly and then flushes immediately will have
> its in-memory changes revert whilst records are written to disk.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
