David Gammon created KAFKA-15116:
------------------------------------
Summary: Kafka Streams processing blocked during rebalance
Key: KAFKA-15116
URL: https://issues.apache.org/jira/browse/KAFKA-15116
Project: Kafka
Issue Type: Bug
Components: streams
Affects Versions: 3.5.0
Reporter: David Gammon
We have a Kafka Streams application that simply takes a messages, processes it
and then produces an event out the other side. The complexity is that there is
a requirement that all events with the same partition key must be committed
before the next message is processed.
This works most of the time flawlessly but we have started to see problems
during deployments where the first message blocks the second message during a
rebalance because the first message isn’t committed before the second message
is processed. This ultimately results in transactions timing out and more
rebalancing.
We’ve tried lots of configuration to get the behaviour we require with no luck.
We’ve now put in a temporary fix so that Kafka Streams works with our framework
but it feels like this might be a missing feature or potentially a bug.
+Example+
Given:
* We have two messages (InA and InB).
* Both messages have the same partition key.
* A rebalance is in progress so streams is no longer able to commit.
When:
# Message InA -> processor -> OutA (not committed)
# Message InB -> processor -> blocked because #1 has not been committed
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
