[
https://issues.apache.org/jira/browse/KAFKA-20416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18109931#comment-18109931
]
Bill Bejeck commented on KAFKA-20416:
-------------------------------------
[~fabianbell] - ok that makes sense. I have a PR for this issue and I'll ping
you here when it's posted
> RocksDB loses entries during broker patches.
> --------------------------------------------
>
> Key: KAFKA-20416
> URL: https://issues.apache.org/jira/browse/KAFKA-20416
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 3.9.1
> Environment: Broker: MSK with kafka.m7g.2xlarge instances
> Application: EKS with m7i.2xlarge instances and Bottlerocket OS 1.57.0
> Reporter: Fabian Bell
> Priority: Major
> Attachments: logs
>
>
> h2. Problem:
> We discovered a strange behaviour on our production environment. We use a
> KTable to look up data from a topic we write to.
>
> {code:java}
> builder.table(topicName, Consumed.with(keySerde, valueSerde),
> Materialized.as(storeName)) {code}
>
> When we access the store in the processor, we observed that the store
> returned null values for keys that have non-null entries in the topic that
> backs the KTable after an MSK security patch. We never tombstone an entry in
> our topic nor have a delete retention activated.
> This only happens for some of our instances.
> We see the following stream logs:
>
> {code:java}
> Committing task(s) 0_14 failed.
> Detected the states of tasks [0_14] are corrupted. Will close the task as
> dirty and re-create and bootstrap from scratch.
> Active task(s) got corrupted. Triggering a rebalance.
> End offset for changelog our-topic-14 initialized as 16596290.
> Restoration in progress for 1 partitions. {our-topic-14: position=0,
> end=16596290, totalRestored=0}
> State transition from RUNNING to PARTITIONS_REVOKED
> No followup rebalance was requested, resetting the rebalance schedule.
> partition revocation took 80 ms.
> State transition from PARTITIONS_REVOKED to PARTITIONS_ASSIGNED
> State transition from PARTITIONS_ASSIGNED to RUNNING {code}
>
> This all happens within a few seconds, and the `Restoration in progress ...`
> log is the only one we can see. A full restoration usually takes like 30 min.
> The error message of the commit failure is
> {code:java}
> o.a.k.c.e.TimeoutException: Timeout expired after 60000ms while awaiting
> AddOffsetsToTxn {code}
> We can fix this situation by clearing the state directory and forcing a full
> restoration.
> h2. Context:
> Each instance has its own persistent state directory. The configured state
> directory does not change.
> Processing Guarantee: exactly_once_v2
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
