[ https://issues.apache.org/jira/browse/KAFKA-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777571#comment-16777571 ]
Ziming Dong commented on KAFKA-7967: ------------------------------------ [~guozhang], since the value rollback to an old value which is several days ago, and our data is not small per partition, it should not be an cache problem? Also, [https://github.com/apache/kafka/pull/6191] says that [https://github.com/apache/kafka/pull/4331] fixed KV store, but we met this issue on both KV store and windowed KV store. Let's upgrade to [https://github.com/apache/kafka/releases/tag/2.2.0-rc0] to see what will happen since both KAFKA-7652 and KAFKA-7672 commits are included.. > Kafka Streams: some values in statestore rollback to old value > -------------------------------------------------------------- > > Key: KAFKA-7967 > URL: https://issues.apache.org/jira/browse/KAFKA-7967 > Project: Kafka > Issue Type: Bug > Components: streams > Affects Versions: 2.1.0 > Reporter: Ziming Dong > Priority: Critical > > We are using kafka streams 2.1.0, we use both persistentKeyValueStore > statestore and persistentWindowStore statestore. We found sometimes both > types of statestore could `fetch` old values instead of newly updated values. > We didn't find any logs except INFO level logs, no instance restart in the > period, also there is no rebalance log which indicates it's not a rebalance > bug. The bug happened no more than one time each week, but many records were > affected each time, and we didn't find a way to reproduce it manually. > For example, the issue may happen like this, note the changelog contains all > the `update`: > # got value 1 from key 1 > # update value 2 to key 1 > # got value 2 from key 1 > # update value 3 to key 1 > # got value 1 from key 1(something wrong!!) > # update value 2 to key 1 > there is only one type log as follow > > {code:java} > 2019-02-19x14:20:00x xxxxxx INFO > [org.apache.kafka.clients.FetchSessionHandler] > [xxxxxxx-streams-xx-xxx-xxxx-xxx-xxxxxx-StreamThread-1] [Consumer > clientId=xxxxxxxxx-xxxx-xxx-xxx-xxxx-xxxxx-StreamThread-1-consumer, > groupId=xxxxxxxxx] Node 2 was unable to process the fetch request with > (sessionId=1998942517, epoch=4357): INVALID_FETCH_SESSION_EPOCH. > {code} > -- This message was sent by Atlassian JIRA (v7.6.3#76005)