[
https://issues.apache.org/jira/browse/KAFKA-14172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17645330#comment-17645330
]
John Gray commented on KAFKA-14172:
-----------------------------------
I do hope this bug gets some eyes eventually, it has been pretty devastating
for us. I amend my previous statement that this is related to our brokers
rolling, it actually seems it can happen any time our restore consumers in EOS
apps w/ big state try to restore said big state. Sadly, setting the
acceptable.lag config does not help us because we do not have the extra space
to run standby threads/replicas.
We actually had to resort to dumping our keys to a sqlserver and then querying
for "should this key exist?" when we are pulling keys from our state store. If
the state store returns nothing and sqlserver says it has seen that key before,
we kill the app, which then causes us to pull the state again, which usually
fixes the issue. Something is _very_ wrong with these restore consumers (or I
am doing something horribly wrong in this app, although we never had this
problem before Kafka 3.0.0 or 3.1.0).
> bug: State stores lose state when tasks are reassigned under EOS wit…
> ---------------------------------------------------------------------
>
> Key: KAFKA-14172
> URL: https://issues.apache.org/jira/browse/KAFKA-14172
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 3.1.1
> Reporter: Martin Hørslev
> Priority: Major
>
> h1. State stores lose state when tasks are reassigned under EOS with standby
> replicas and default acceptable lag.
> I have observed that state stores used in a transform step under a Exactly
> Once semantics ends up losing state after a rebalancing event that includes
> reassignment of tasks to previous standby task within the acceptable standby
> lag.
>
> The problem is reproduceable and an integration test have been created to
> showcase the [issue|https://github.com/apache/kafka/pull/12540].
> A detailed description of the observed issue is provided
> [here|https://github.com/apache/kafka/pull/12540/files?short_path=3ca480e#diff-3ca480ef093a1faa18912e1ebc679be492b341147b96d7a85bda59911228ef45]
> Similar issues have been observed and reported to StackOverflow for example
> [here|https://stackoverflow.com/questions/69038181/kafka-streams-aggregation-data-loss-between-instance-restarts-and-rebalances].
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
