Greg Harris created KAFKA-15575: ----------------------------------- Summary: Prevent Connectors from exceeding tasks.max configuration Key: KAFKA-15575 URL: https://issues.apache.org/jira/browse/KAFKA-15575 Project: Kafka Issue Type: Task Components: KafkaConnect Reporter: Greg Harris
The Connector::taskConfigs(int maxTasks) function is used by Connectors to enumerate tasks configurations. This takes an argument which comes from the tasks.max connector config. This is the Javadoc for that method: {noformat} /** * Returns a set of configurations for Tasks based on the current configuration, * producing at most {@code maxTasks} configurations. * * @param maxTasks maximum number of configurations to generate * @return configurations for Tasks */ public abstract List<Map<String, String>> taskConfigs(int maxTasks); {noformat} This includes the constraint that the number of tasks is at most maxTasks, but this constraint is not enforced by the framework. We should begin enforcing this constraint by dropping configs that exceed the limit, and logging a warning. For sink connectors this should harmlessly rebalance the consumer subscriptions onto the remaining tasks. For source connectors that distribute their work via task configs, this may result in an interruption in data transfer. -- This message was sent by Atlassian Jira (v8.20.10#820010)