Matthias J. Sax created KAFKA-4509:
--------------------------------------
Summary: Task reusage on rebalance fails for threads on same host
Key: KAFKA-4509
URL: https://issues.apache.org/jira/browse/KAFKA-4509
Project: Kafka
Issue Type: Bug
Components: streams
Reporter: Matthias J. Sax
Assignee: Matthias J. Sax
In https://issues.apache.org/jira/browse/KAFKA-3559 task reusage on rebalance
was introduces as a performance optimization. Instead of closing a task on
rebalance (ie, {{onPartitionsRevoked()}}, it only get's suspended for a
potential reuse in {{onPartitionsAssigned()}}. Only if a task cannot be reused,
it will eventually get closed in {{onPartitionsAssigned()}}.
This mechanism can fail, if multiple {{StreamThreads}} run in the same host
(same or different JVM). The scenario is as follows:
- assume 2 running threads A and B
- assume 3 tasks t1, t2, t3
- assignment: A-(t1,t2) and B-(t3)
- on the same host, a new single threaded Stream application (same app-id)
gets started (thread C)
- on rebalance, t2 (could also be t1 -- does not matter) will be moved from A
to C
- as assignment is only sticky base on an heurictic t1 can sometimes be
assigned to B, too -- and t3 get's assigned to A (thre is a race condition if
this "task flipping" happens or not)
- on revoke, A will suspend task t1 and t2 (not releasing any locks)
- on assign
- A tries to create t3 but as B did not release it yet, A dies with an
"cannot get lock" exception
- B tries to create t1 but as A did not release it yet, B dies with an
"cannot get lock" exception
- as A and B trie to create the task first, this will always fail if task
flipping happened
- C tries to create t2 but A did not release t2 lock yet (race condition)
and C dies with an exception (this could even happen without "task flipping"
between A and B)
We want to fix this, by:
# first release unassigned suspended tasks in {{onPartitionsAssignment()}},
and afterward create new tasks (this fixes the "task flipping" issue)
# use a "backoff and retry mechanism" if a task cannot be created (to handle
release-create race condition between different threads)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
