[
https://issues.apache.org/jira/browse/KAFKA-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ziming Dong updated KAFKA-7967:
-------------------------------
Description:
We are using kafka streams 2.1.0, we use both persistentKeyValueStore
statestore and persistentWindowStore statestore. We found sometimes both types
of statestore could `fetch` old values instead of newly updated values. We
didn't find any logs except INFO level logs, no instance restart in the period,
also there is no rebalance log which indicates it's not a rebalance bug. The
bug happened no more than one time each week, but many records were affected
each time, and we didn't find a way to reproduce it manually.
For example, the issue may happen like this:
# got value 1 from key 1
# update value 2 to key 1
# got value 2 from key 1
# update value 3 to key 1
# got value 1 from key 1(something wrong!!)
# update value 2 to key 1
there is only one type log as follow
{code:java}
2019-02-19x14:20:00x xxxxxx INFO [org.apache.kafka.clients.FetchSessionHandler]
[xxxxxxx-streams-xx-xxx-xxxx-xxx-xxxxxx-StreamThread-1] [Consumer
clientId=xxxxxxxxx-xxxx-xxx-xxx-xxxx-xxxxx-StreamThread-1-consumer,
groupId=xxxxxxxxx] Node 2 was unable to process the fetch request with
(sessionId=1998942517, epoch=4357): INVALID_FETCH_SESSION_EPOCH.
{code}
was:
We are using kafka streams 2.1.0, we use both persistentKeyValueStore
statestore and persistentWindowStore statestore. We found sometimes both types
of statestore could `fetch` old values instead of newly updated values. We
didn't find any logs except INFO level logs, also there is no rebalance log
which indicates it's not a rebalance bug. The bug happened no more than one
time each week, but many records were affected each time, and we didn't find a
way to reproduce it manually.
For example, the issue may happen like this:
# got value 1 from key 1
# update value 2 to key 1
# got value 2 from key 1
# update value 3 to key 1
# got value 1 from key 1(something wrong!!)
# update value 2 to key 1
there is only one type log as follow
{code:java}
2019-02-19x14:20:00x xxxxxx INFO [org.apache.kafka.clients.FetchSessionHandler]
[xxxxxxx-streams-xx-xxx-xxxx-xxx-xxxxxx-StreamThread-1] [Consumer
clientId=xxxxxxxxx-xxxx-xxx-xxx-xxxx-xxxxx-StreamThread-1-consumer,
groupId=xxxxxxxxx] Node 2 was unable to process the fetch request with
(sessionId=1998942517, epoch=4357): INVALID_FETCH_SESSION_EPOCH.
{code}
> Kafka Streams: some values in statestore rollback to old value
> --------------------------------------------------------------
>
> Key: KAFKA-7967
> URL: https://issues.apache.org/jira/browse/KAFKA-7967
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 2.1.0
> Reporter: Ziming Dong
> Priority: Critical
>
> We are using kafka streams 2.1.0, we use both persistentKeyValueStore
> statestore and persistentWindowStore statestore. We found sometimes both
> types of statestore could `fetch` old values instead of newly updated values.
> We didn't find any logs except INFO level logs, no instance restart in the
> period, also there is no rebalance log which indicates it's not a rebalance
> bug. The bug happened no more than one time each week, but many records were
> affected each time, and we didn't find a way to reproduce it manually.
> For example, the issue may happen like this:
> # got value 1 from key 1
> # update value 2 to key 1
> # got value 2 from key 1
> # update value 3 to key 1
> # got value 1 from key 1(something wrong!!)
> # update value 2 to key 1
> there is only one type log as follow
>
> {code:java}
> 2019-02-19x14:20:00x xxxxxx INFO
> [org.apache.kafka.clients.FetchSessionHandler]
> [xxxxxxx-streams-xx-xxx-xxxx-xxx-xxxxxx-StreamThread-1] [Consumer
> clientId=xxxxxxxxx-xxxx-xxx-xxx-xxxx-xxxxx-StreamThread-1-consumer,
> groupId=xxxxxxxxx] Node 2 was unable to process the fetch request with
> (sessionId=1998942517, epoch=4357): INVALID_FETCH_SESSION_EPOCH.
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
