[
https://issues.apache.org/jira/browse/KAFKA-15116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17738782#comment-17738782
]
Matthias J. Sax commented on KAFKA-15116:
-----------------------------------------
> Message A uses an internal store to store information about the entity. The
> store knows that there is a pending event that is yet to be committed so it
> blocks until it is committed.
Are you saying that this happens in a background thread that you start
yourself? If yes, it a non-supported pattern, and we cannot give any guarantee
about the behavior of the system. If there is no background thread, that
blocking would imply that `StreamThread` blocks (also something you should not
do, as it would imply the that thread drops out of the consumer group after
`max.poll.interval.ms` passed, and thus, how could message B get processed? Or
is this internal store that you mentioned shared across `StreamThreads` (this
would also be an anti-pattern, and we cannot give any guarantee how the system
behalves if you do this).
> The store knows that there is a pending event that is yet to be committed so
> it blocks until it is committed.
I am also wondering what you exactly mean by "committed" (it's a highly
overloaded term, so it would be good to clarify). In Kafka itself, there could
be two meanings: for at-least-once-processing "committing" means to commit the
input topic offsets and mark the input records as processed. For
exaclty-once-processing "committing" means to commit the Kafka TX, ie,
committing the result record into the output topic plus committing the input
topic offset to mark the input records as processed. Not sure which one you
mean, or if you actually refer to some mechanism to commit into your internal
store?
I guess, I still don't understand your overall end-to-end workflow of your
program.
> Kafka Streams processing blocked during rebalance
> -------------------------------------------------
>
> Key: KAFKA-15116
> URL: https://issues.apache.org/jira/browse/KAFKA-15116
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 3.5.0
> Reporter: David Gammon
> Priority: Major
>
> We have a Kafka Streams application that simply takes a messages, processes
> it and then produces an event out the other side. The complexity is that
> there is a requirement that all events with the same partition key must be
> committed before the next message is processed.
> This works most of the time flawlessly but we have started to see problems
> during deployments where the first message blocks the second message during a
> rebalance because the first message isn’t committed before the second message
> is processed. This ultimately results in transactions timing out and more
> rebalancing.
> We’ve tried lots of configuration to get the behaviour we require with no
> luck. We’ve now put in a temporary fix so that Kafka Streams works with our
> framework but it feels like this might be a missing feature or potentially a
> bug.
> +Example+
> Given:
> * We have two messages (InA and InB).
> * Both messages have the same partition key.
> * A rebalance is in progress so streams is no longer able to commit.
> When:
> # Message InA -> processor -> OutA (not committed)
> # Message InB -> processor -> blocked because #1 has not been committed
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
