[
https://issues.apache.org/jira/browse/KAFKA-15520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17770635#comment-17770635
]
A. Sophie Blee-Goldman commented on KAFKA-15520:
------------------------------------------------
{quote}Also added
props.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG,
CooperativeStickyAssignor.class.getName());
to enable CooperativeStickyAssignor
{quote}
To clarify, Streams will always override and ignore this setting because it
needs to plug in the StreamsPartitionAssignor – so you don't need this, at
least not for any Streams applications. It should use cooperative rebalancing
by default as of version 2.4 and above.
I guess the one thing to watch out for is that cooperative rebalancing will not
be enabled if you upgrade from a lower version and forget/skip the step to
remove the StreamsConfig.UPGRADE_FROM property. So just make sure that's not
being set anywhere (at least, not set to a version that's lower than 2.4)
> Kafka Streams Stateful Aggregation Rebalancing causing processing to pause on
> all partitions
> --------------------------------------------------------------------------------------------
>
> Key: KAFKA-15520
> URL: https://issues.apache.org/jira/browse/KAFKA-15520
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 2.6.2
> Reporter: Rohit Bobade
> Priority: Major
>
> Kafka broker version: 2.8.0 Kafka Streams client version: 2.6.2
> I am running kafka streams stateful aggregations on K8s statefulset with
> persistent volume attached to each pod. I have also specified
> props.put(ConsumerConfig.GROUP_INSTANCE_ID_CONFIG, podName);
> which makes sure it gets the sticky partition assignment.
> Enabled standby replica -
> props.put(StreamsConfig.NUM_STANDBY_REPLICAS_CONFIG, 1);
> and set props.put(StreamsConfig.ACCEPTABLE_RECOVERY_LAG_CONFIG, "0");
> However, I'm seeing that when pods restart - it triggers rebalances and
> causes processing to be paused on all pods till the rebalance and state
> restore is in progress.
> My understanding is that even if there is a rebalance - only the partitions
> that should be moved around will be restored in a cooperative way and not
> pause all the processing. Also, it should failover to standby replica in this
> case and avoid state restoring on other pods.
> I have increased session timeout to 480 seconds and max poll interval to 15
> mins to minimize rebalances.
> Also added
> props.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG,
> CooperativeStickyAssignor.class.getName());
> to enable CooperativeStickyAssignor
> could someone please help if I'm missing something?
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
