[
https://issues.apache.org/jira/browse/KAFKA-5505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16317333#comment-16317333
]
Randall Hauch commented on KAFKA-5505:
--------------------------------------
We'll likely change/evolve Connect's rebalance subprotocol to enable
implementing this improvement. See also KAFKA-6433 for another improvement that
will require changes to the metadata used in the subprotocol.
> Connect: Do not restart connector and existing tasks on task-set change
> -----------------------------------------------------------------------
>
> Key: KAFKA-5505
> URL: https://issues.apache.org/jira/browse/KAFKA-5505
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
> Affects Versions: 0.10.2.1
> Reporter: Per Steffensen
>
> I am writing a connector with a frequently changing task-set. It is really
> not working very well, because the connector and all existing tasks are
> restarted when the set of tasks changes. E.g. if the connector is running
> with 10 tasks, and an additional task is needed, the connector itself and all
> 10 existing tasks are restarted, just to make the 11th task run also. My
> tasks have a fairly heavy initialization, making it extra annoying. I would
> like to see a change, introducing a "mode", where only new/deleted tasks are
> started/stopped when notifying the system that the set of tasks changed
> (calling context.requestTaskReconfiguration() - or something similar).
> Discussed this issue a little on [email protected] in the thread "Kafka
> Connect: To much restarting with a SourceConnector with dynamic set of tasks"
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
