[
https://issues.apache.org/jira/browse/KAFKA-15116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17736637#comment-17736637
]
Matthias J. Sax commented on KAFKA-15116:
-----------------------------------------
> The complexity is that there is a requirement that all events with the same
> partition key must be committed before the next message is processed.
Not sure if I understand this requirement. Can you elaborate? The input is an
infinite stream. So if you get, let's say 3 messages all with key A, how do you
know that there is no 4th message with key A? – Also, in Kafka Streams you
cannot really control when a commit happens to begin with.
> where the first message blocks the second message during a rebalance because
> the first message isn’t committed before the second message is processed
Also not sure what this means? If a rebalance is triggered, all pending
messages will be flushed out, the offset will be committed.
> This ultimately results in transactions timing out and more rebalancing.
Kafka Streams manages transaction under the hood for you. So you don't know
when a TX starts or ends. How can you reason about it?
> We’ve now put in a temporary fix
Can you give more details?
> A rebalance is in progress so streams is no longer able to commit.
If a rebalance is triggered, KS should first commit before the rebalance goes
into "in progress" state – and thus, it should not be necessary to commit (it
was already done).
> Kafka Streams processing blocked during rebalance
> -------------------------------------------------
>
> Key: KAFKA-15116
> URL: https://issues.apache.org/jira/browse/KAFKA-15116
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 3.5.0
> Reporter: David Gammon
> Priority: Major
>
> We have a Kafka Streams application that simply takes a messages, processes
> it and then produces an event out the other side. The complexity is that
> there is a requirement that all events with the same partition key must be
> committed before the next message is processed.
> This works most of the time flawlessly but we have started to see problems
> during deployments where the first message blocks the second message during a
> rebalance because the first message isn’t committed before the second message
> is processed. This ultimately results in transactions timing out and more
> rebalancing.
> We’ve tried lots of configuration to get the behaviour we require with no
> luck. We’ve now put in a temporary fix so that Kafka Streams works with our
> framework but it feels like this might be a missing feature or potentially a
> bug.
> +Example+
> Given:
> * We have two messages (InA and InB).
> * Both messages have the same partition key.
> * A rebalance is in progress so streams is no longer able to commit.
> When:
> # Message InA -> processor -> OutA (not committed)
> # Message InB -> processor -> blocked because #1 has not been committed
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
