[ https://issues.apache.org/jira/browse/KAFKA-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16773752#comment-16773752 ]
Ziming Dong commented on KAFKA-7967: ------------------------------------ {code:java} final StreamsBuilder builder = new StreamsBuilder(); final StoreBuilder<WindowStore<Integer, Long>> store_1 = Stores.windowStoreBuilder( Stores.persistentWindowStore("store_1", xx, xx, false), Serdes.Integer(), Serdes.Long()); builder.addStateStore(store_1); final StoreBuilder<WindowStore<Integer, Long>> store_2 = Stores.windowStoreBuilder( Stores.persistentWindowStore("store_2", xx, xx, false), Serdes.Integer(), Serdes.Long()); builder.addStateStore(store_2); final StoreBuilder<WindowStore<Integer, Long>> store_3 = Stores.windowStoreBuilder( Stores.persistentWindowStore("store_3", xx, xx, false), Serdes.Integer(), Serdes.Long()); builder.addStateStore(store_3); final StoreBuilder<KeyValueStore<Integer, Long>> store_4 = Stores.keyValueStoreBuilder( Stores.persistentKeyValueStore("store_4"), Serdes.Integer(), Serdes.Long()); builder.addStateStore(store_4);{code} and in the `transformer`, for each record, we just get 4 values and update 4 values each time. We found roll back issue happened two times, in the first time(last week), only store_1 affected, in the second time(this week), only store_4 affected. We have exactly once enabled. DEBUG log not enabled. We thought cache is disabled by default? > Kafka Streams: some values in statestore rollback to old value > -------------------------------------------------------------- > > Key: KAFKA-7967 > URL: https://issues.apache.org/jira/browse/KAFKA-7967 > Project: Kafka > Issue Type: Bug > Affects Versions: 2.1.0 > Reporter: Ziming Dong > Priority: Critical > > We are using kafka streams 2.1.0, we use both persistentKeyValueStore > statestore and persistentWindowStore statestore. We found sometimes both > types of statestore could `fetch` old values instead of newly updated values. > We didn't find any logs except INFO level logs, no instance restart in the > period, also there is no rebalance log which indicates it's not a rebalance > bug. The bug happened no more than one time each week, but many records were > affected each time, and we didn't find a way to reproduce it manually. > For example, the issue may happen like this, note the changelog contains all > the `update`: > # got value 1 from key 1 > # update value 2 to key 1 > # got value 2 from key 1 > # update value 3 to key 1 > # got value 1 from key 1(something wrong!!) > # update value 2 to key 1 > there is only one type log as follow > > {code:java} > 2019-02-19x14:20:00x xxxxxx INFO > [org.apache.kafka.clients.FetchSessionHandler] > [xxxxxxx-streams-xx-xxx-xxxx-xxx-xxxxxx-StreamThread-1] [Consumer > clientId=xxxxxxxxx-xxxx-xxx-xxx-xxxx-xxxxx-StreamThread-1-consumer, > groupId=xxxxxxxxx] Node 2 was unable to process the fetch request with > (sessionId=1998942517, epoch=4357): INVALID_FETCH_SESSION_EPOCH. > {code} > -- This message was sent by Atlassian JIRA (v7.6.3#76005)