Bradley Peterson created KAFKA-10678:
----------------------------------------
Summary: Re-deploying Streams app causes rebalance and task
migration
Key: KAFKA-10678
URL: https://issues.apache.org/jira/browse/KAFKA-10678
Project: Kafka
Issue Type: Bug
Components: streams
Affects Versions: 2.6.0, 2.6.1
Reporter: Bradley Peterson
Attachments: after, before, broker
Re-deploying our Streams app causes a rebalance, even when using static group
membership. Worse, the rebalance creates standby tasks, even when the previous
task assignment was balanced and stable.
Our app is currently using Streams 2.6.1-SNAPSHOT (due to [KAFKA-10633]) but we
saw the same behavior in 2.6.0. The app runs on 4 EC2 instances, each with 4
streams threads, and data stored on persistent EBS volumes.. During a redeploy,
all EC2 instances are stopped, new instances are launched, and the EBS volumes
are attached to the new instances. We do not use interactive queries.
{{session.timeout.ms}} is set to 30 minutes, and the deployment finishes well
under that. {{num.standby.replicas}} is 0.
h2. Expected Behavior
Given a stable and balanced task assignment prior to deploying, we expect to
see the same task assignment after deploying. Even if a rebalance is triggered,
we do not expect to see new standby tasks.
h2. Observed Behavior
Attached are the "Assigned tasks to clients" log lines from before and after
deploying. The "before" is from over 24 hours ago, the task assignment is well
balanced and "Finished stable assignment of tasks, no followup rebalances
required." is logged. The "after" log lines show the same assignment of active
tasks, but some additional standby tasks. There are additional log lines about
adding and removing active tasks, which I don't quite understand.
I've also included logs from the broker showing the rebalance was triggered for
"Updating metadata".
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
