[
https://issues.apache.org/jira/browse/KAFKA-20416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18116784#comment-18116784
]
Matthias J. Sax commented on KAFKA-20416:
-----------------------------------------
[~fabianbell] – you are saying "the current supported MSK version is only
4.2.x. Thereby, it does not solve our problem."
Not sure how to read this? Are you using 4.2 or an older version like 3.9
broker side? It sound like you don't?
What we are saying it: there is a broker side bug in 3.9 and older, which is
fixed in 4.0 an newer. And this broker side bug is triggering the issue in KS
you observe (which is not really a bug in KS, but a "fall out" of the broker
bug).
So if you are on MSK/3.9 and you upgrade to MSK/4.0 or newer, the issue should
go away. No broker bug implies that KS does do the right thing.
Can you confirm what broker version you are running?
> RocksDB loses entries during broker patches.
> --------------------------------------------
>
> Key: KAFKA-20416
> URL: https://issues.apache.org/jira/browse/KAFKA-20416
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 3.9.1
> Environment: Broker: MSK with kafka.m7g.2xlarge instances
> Application: EKS with m7i.2xlarge instances and Bottlerocket OS 1.57.0
> Reporter: Fabian Bell
> Assignee: Bill Bejeck
> Priority: Major
> Attachments: logs
>
>
> h2. Problem:
> We discovered a strange behaviour on our production environment. We use a
> KTable to look up data from a topic we write to.
>
> {code:java}
> builder.table(topicName, Consumed.with(keySerde, valueSerde),
> Materialized.as(storeName)) {code}
>
> When we access the store in the processor, we observed that the store
> returned null values for keys that have non-null entries in the topic that
> backs the KTable after an MSK security patch. We never tombstone an entry in
> our topic nor have a delete retention activated.
> This only happens for some of our instances.
> We see the following stream logs:
>
> {code:java}
> Committing task(s) 0_14 failed.
> Detected the states of tasks [0_14] are corrupted. Will close the task as
> dirty and re-create and bootstrap from scratch.
> Active task(s) got corrupted. Triggering a rebalance.
> End offset for changelog our-topic-14 initialized as 16596290.
> Restoration in progress for 1 partitions. {our-topic-14: position=0,
> end=16596290, totalRestored=0}
> State transition from RUNNING to PARTITIONS_REVOKED
> No followup rebalance was requested, resetting the rebalance schedule.
> partition revocation took 80 ms.
> State transition from PARTITIONS_REVOKED to PARTITIONS_ASSIGNED
> State transition from PARTITIONS_ASSIGNED to RUNNING {code}
>
> This all happens within a few seconds, and the `Restoration in progress ...`
> log is the only one we can see. A full restoration usually takes like 30 min.
> The error message of the commit failure is
> {code:java}
> o.a.k.c.e.TimeoutException: Timeout expired after 60000ms while awaiting
> AddOffsetsToTxn {code}
> We can fix this situation by clearing the state directory and forcing a full
> restoration.
> h2. Context:
> Each instance has its own persistent state directory. The configured state
> directory does not change.
> Processing Guarantee: exactly_once_v2
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
