[jira] [Commented] (KAFKA-5505) Connect: Do not restart connector and existing tasks on task-set change
[ https://issues.apache.org/jira/browse/KAFKA-5505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16841884#comment-16841884 ] ASF GitHub Bot commented on KAFKA-5505: --- rhauch commented on pull request #6363: KAFKA-5505: Incremental cooperative rebalancing in Connect (KIP-415) URL: https://github.com/apache/kafka/pull/6363 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Connect: Do not restart connector and existing tasks on task-set change > --- > > Key: KAFKA-5505 > URL: https://issues.apache.org/jira/browse/KAFKA-5505 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Affects Versions: 0.10.2.1 >Reporter: Per Steffensen >Assignee: Konstantine Karantasis >Priority: Major > > I am writing a connector with a frequently changing task-set. It is really > not working very well, because the connector and all existing tasks are > restarted when the set of tasks changes. E.g. if the connector is running > with 10 tasks, and an additional task is needed, the connector itself and all > 10 existing tasks are restarted, just to make the 11th task run also. My > tasks have a fairly heavy initialization, making it extra annoying. I would > like to see a change, introducing a "mode", where only new/deleted tasks are > started/stopped when notifying the system that the set of tasks changed > (calling context.requestTaskReconfiguration() - or something similar). > Discussed this issue a little on d...@kafka.apache.org in the thread "Kafka > Connect: To much restarting with a SourceConnector with dynamic set of tasks" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-5505) Connect: Do not restart connector and existing tasks on task-set change
[ https://issues.apache.org/jira/browse/KAFKA-5505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16782988#comment-16782988 ] ASF GitHub Bot commented on KAFKA-5505: --- kkonstantine commented on pull request #6363: KAFKA-5505: Incremental cooperative rebalancing in Connect (KIP-415) URL: https://github.com/apache/kafka/pull/6363 ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Connect: Do not restart connector and existing tasks on task-set change > --- > > Key: KAFKA-5505 > URL: https://issues.apache.org/jira/browse/KAFKA-5505 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Affects Versions: 0.10.2.1 >Reporter: Per Steffensen >Assignee: Konstantine Karantasis >Priority: Major > > I am writing a connector with a frequently changing task-set. It is really > not working very well, because the connector and all existing tasks are > restarted when the set of tasks changes. E.g. if the connector is running > with 10 tasks, and an additional task is needed, the connector itself and all > 10 existing tasks are restarted, just to make the 11th task run also. My > tasks have a fairly heavy initialization, making it extra annoying. I would > like to see a change, introducing a "mode", where only new/deleted tasks are > started/stopped when notifying the system that the set of tasks changed > (calling context.requestTaskReconfiguration() - or something similar). > Discussed this issue a little on d...@kafka.apache.org in the thread "Kafka > Connect: To much restarting with a SourceConnector with dynamic set of tasks" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-5505) Connect: Do not restart connector and existing tasks on task-set change
[ https://issues.apache.org/jira/browse/KAFKA-5505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16609666#comment-16609666 ] satyanarayan komandur commented on KAFKA-5505: -- Is there any update on the re balance improvement. When i am running a single worker node (in distributed mode in dev environment), i still see re-balancing going on resulting in restart of connectors/tasks. Is'nt re-balance really a scenario where multiple Worker nodes are involved. > Connect: Do not restart connector and existing tasks on task-set change > --- > > Key: KAFKA-5505 > URL: https://issues.apache.org/jira/browse/KAFKA-5505 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Affects Versions: 0.10.2.1 >Reporter: Per Steffensen >Priority: Major > > I am writing a connector with a frequently changing task-set. It is really > not working very well, because the connector and all existing tasks are > restarted when the set of tasks changes. E.g. if the connector is running > with 10 tasks, and an additional task is needed, the connector itself and all > 10 existing tasks are restarted, just to make the 11th task run also. My > tasks have a fairly heavy initialization, making it extra annoying. I would > like to see a change, introducing a "mode", where only new/deleted tasks are > started/stopped when notifying the system that the set of tasks changed > (calling context.requestTaskReconfiguration() - or something similar). > Discussed this issue a little on d...@kafka.apache.org in the thread "Kafka > Connect: To much restarting with a SourceConnector with dynamic set of tasks" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-5505) Connect: Do not restart connector and existing tasks on task-set change
[ https://issues.apache.org/jira/browse/KAFKA-5505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16317333#comment-16317333 ] Randall Hauch commented on KAFKA-5505: -- We'll likely change/evolve Connect's rebalance subprotocol to enable implementing this improvement. See also KAFKA-6433 for another improvement that will require changes to the metadata used in the subprotocol. > Connect: Do not restart connector and existing tasks on task-set change > --- > > Key: KAFKA-5505 > URL: https://issues.apache.org/jira/browse/KAFKA-5505 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Affects Versions: 0.10.2.1 >Reporter: Per Steffensen > > I am writing a connector with a frequently changing task-set. It is really > not working very well, because the connector and all existing tasks are > restarted when the set of tasks changes. E.g. if the connector is running > with 10 tasks, and an additional task is needed, the connector itself and all > 10 existing tasks are restarted, just to make the 11th task run also. My > tasks have a fairly heavy initialization, making it extra annoying. I would > like to see a change, introducing a "mode", where only new/deleted tasks are > started/stopped when notifying the system that the set of tasks changed > (calling context.requestTaskReconfiguration() - or something similar). > Discussed this issue a little on d...@kafka.apache.org in the thread "Kafka > Connect: To much restarting with a SourceConnector with dynamic set of tasks" -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-5505) Connect: Do not restart connector and existing tasks on task-set change
[ https://issues.apache.org/jira/browse/KAFKA-5505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16256037#comment-16256037 ] Gaurav Nanda commented on KAFKA-5505: - Yea, the root cause for both our problems seems to be the same, kafka connect is not able to add/remove/reconfigure tasks without restarting all the tasks running on the cluster. > Connect: Do not restart connector and existing tasks on task-set change > --- > > Key: KAFKA-5505 > URL: https://issues.apache.org/jira/browse/KAFKA-5505 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Affects Versions: 0.10.2.1 >Reporter: Per Steffensen > > I am writing a connector with a frequently changing task-set. It is really > not working very well, because the connector and all existing tasks are > restarted when the set of tasks changes. E.g. if the connector is running > with 10 tasks, and an additional task is needed, the connector itself and all > 10 existing tasks are restarted, just to make the 11th task run also. My > tasks have a fairly heavy initialization, making it extra annoying. I would > like to see a change, introducing a "mode", where only new/deleted tasks are > started/stopped when notifying the system that the set of tasks changed > (calling context.requestTaskReconfiguration() - or something similar). > Discussed this issue a little on d...@kafka.apache.org in the thread "Kafka > Connect: To much restarting with a SourceConnector with dynamic set of tasks" -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-5505) Connect: Do not restart connector and existing tasks on task-set change
[
https://issues.apache.org/jira/browse/KAFKA-5505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16224054#comment-16224054
]
Per Steffensen commented on KAFKA-5505:
---
[~gauravnanda] it sounds like your problem is slightly different than the one
actually reported here. I reported that it is annoying that all tasks and the
connector itself is restarted when an existing connector changes its set of
tasks (by calling {{context.requestTaskReconfiguration()}}). It seems like you
are annoyed that all existing tasks and connectors are restarted when you
create a new connector. While the problems differ slightly, the solution is
probably "kinda the same", so maybe we should just make this ticket
(KAFKA-5505) cover both?
> Connect: Do not restart connector and existing tasks on task-set change
> ---
>
> Key: KAFKA-5505
> URL: https://issues.apache.org/jira/browse/KAFKA-5505
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
>Affects Versions: 0.10.2.1
>Reporter: Per Steffensen
>
> I am writing a connector with a frequently changing task-set. It is really
> not working very well, because the connector and all existing tasks are
> restarted when the set of tasks changes. E.g. if the connector is running
> with 10 tasks, and an additional task is needed, the connector itself and all
> 10 existing tasks are restarted, just to make the 11th task run also. My
> tasks have a fairly heavy initialization, making it extra annoying. I would
> like to see a change, introducing a "mode", where only new/deleted tasks are
> started/stopped when notifying the system that the set of tasks changed
> (calling context.requestTaskReconfiguration() - or something similar).
> Discussed this issue a little on d...@kafka.apache.org in the thread "Kafka
> Connect: To much restarting with a SourceConnector with dynamic set of tasks"
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KAFKA-5505) Connect: Do not restart connector and existing tasks on task-set change
[ https://issues.apache.org/jira/browse/KAFKA-5505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16222835#comment-16222835 ] Gaurav Nanda commented on KAFKA-5505: - We also have been facing this issue, we have a bunch of source connectors running on a connect cluster in production, everytime we create a new connector all the existing connectors are restarted for rebalancing. This disrupts the data loading process for some time, more then number of connectors and tasks, longer it takes for the process to get back up and running. We should be able to avoid restarting all existing connectors if a new one is added to the cluster. The only solution for us is to have a new cluster created dynamically when we need a new connector. Which is a clunky solution ... I have talked to some people in meetups and have found more people who are affected by this. I would really recommend fixing this. > Connect: Do not restart connector and existing tasks on task-set change > --- > > Key: KAFKA-5505 > URL: https://issues.apache.org/jira/browse/KAFKA-5505 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Affects Versions: 0.10.2.1 >Reporter: Per Steffensen > > I am writing a connector with a frequently changing task-set. It is really > not working very well, because the connector and all existing tasks are > restarted when the set of tasks changes. E.g. if the connector is running > with 10 tasks, and an additional task is needed, the connector itself and all > 10 existing tasks are restarted, just to make the 11th task run also. My > tasks have a fairly heavy initialization, making it extra annoying. I would > like to see a change, introducing a "mode", where only new/deleted tasks are > started/stopped when notifying the system that the set of tasks changed > (calling context.requestTaskReconfiguration() - or something similar). > Discussed this issue a little on d...@kafka.apache.org in the thread "Kafka > Connect: To much restarting with a SourceConnector with dynamic set of tasks" -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-5505) Connect: Do not restart connector and existing tasks on task-set change
[ https://issues.apache.org/jira/browse/KAFKA-5505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16160845#comment-16160845 ] Per Steffensen commented on KAFKA-5505: --- Just realized that SourceConnector has a initialize method that is given the "existing tasks configuration". That will help me a little. Besides that I was thinking that it may be easier to add more light-weight callbacks to the system via ConnectorContext. Today you only have requestTaskReconfiguration, which I guess tells the system that something changed - anything could have changed: several tasks added, several tasks removed, config change etc. What if we added two additional callbacks on ConnectorContext - addTask and removeTask - used to add and remove tasks respectively. That way the system does not have to do a complicated check of what changed and act appropriately (do the minimum required actions) as it probably has to with requestTaskReconfiguration today. We are leaving more of the detailed management of tasks to the concrete connector itself (if it wants to). Will that be easier? Guess, since this is new callbacks, that it will be easier to guarantee backwards compatibility on requestTaskReconfiguration? > Connect: Do not restart connector and existing tasks on task-set change > --- > > Key: KAFKA-5505 > URL: https://issues.apache.org/jira/browse/KAFKA-5505 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Affects Versions: 0.10.2.1 >Reporter: Per Steffensen > > I am writing a connector with a frequently changing task-set. It is really > not working very well, because the connector and all existing tasks are > restarted when the set of tasks changes. E.g. if the connector is running > with 10 tasks, and an additional task is needed, the connector itself and all > 10 existing tasks are restarted, just to make the 11th task run also. My > tasks have a fairly heavy initialization, making it extra annoying. I would > like to see a change, introducing a "mode", where only new/deleted tasks are > started/stopped when notifying the system that the set of tasks changed > (calling context.requestTaskReconfiguration() - or something similar). > Discussed this issue a little on d...@kafka.apache.org in the thread "Kafka > Connect: To much restarting with a SourceConnector with dynamic set of tasks" -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-5505) Connect: Do not restart connector and existing tasks on task-set change
[ https://issues.apache.org/jira/browse/KAFKA-5505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16159614#comment-16159614 ] Ted Yu commented on KAFKA-5505: --- bq. source-connector out-of-the-box that can copy from a relational database table It would be nice if the connector task(s) from the call(s) to requestTaskReconfiguration can be persisted so that after connector restart, connector can pick them back up. > Connect: Do not restart connector and existing tasks on task-set change > --- > > Key: KAFKA-5505 > URL: https://issues.apache.org/jira/browse/KAFKA-5505 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Affects Versions: 0.10.2.1 >Reporter: Per Steffensen > > I am writing a connector with a frequently changing task-set. It is really > not working very well, because the connector and all existing tasks are > restarted when the set of tasks changes. E.g. if the connector is running > with 10 tasks, and an additional task is needed, the connector itself and all > 10 existing tasks are restarted, just to make the 11th task run also. My > tasks have a fairly heavy initialization, making it extra annoying. I would > like to see a change, introducing a "mode", where only new/deleted tasks are > started/stopped when notifying the system that the set of tasks changed > (calling context.requestTaskReconfiguration() - or something similar). > Discussed this issue a little on d...@kafka.apache.org in the thread "Kafka > Connect: To much restarting with a SourceConnector with dynamic set of tasks" -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-5505) Connect: Do not restart connector and existing tasks on task-set change
[ https://issues.apache.org/jira/browse/KAFKA-5505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16107727#comment-16107727 ] Dan Collins commented on KAFKA-5505: Hi, I wanted to add on to this given we are working around the same behavior, but for a different use case. We have an orchestration service that takes care of wiring sinks out of Kafka using Connect and some of the orchestration scenarios we have require us to setup or modify tens of configurations in Connect at a time. Our current approach to work around this is to retry with backoff and pre-package connector configurations, but it seems like there should be a better way. It's also a bit painful to get a 409 Conflict on GET request for current configurations when tasks are rebalancing especially given we may have multiple orchestration events running simultaneously. The requested change here to only rebalance for new/deleted tasks would be a nice improvement as it'd decrease the initialization time/improve availability. Other thoughts we've had on this would be to expose a new endpoint with bulk api support or a flag on configurations to indicate that new tasks should not immediately trigger a rebalance or may be scheduled in the future. Thanks! -Dan > Connect: Do not restart connector and existing tasks on task-set change > --- > > Key: KAFKA-5505 > URL: https://issues.apache.org/jira/browse/KAFKA-5505 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Affects Versions: 0.10.2.1 >Reporter: Per Steffensen > > I am writing a connector with a frequently changing task-set. It is really > not working very well, because the connector and all existing tasks are > restarted when the set of tasks changes. E.g. if the connector is running > with 10 tasks, and an additional task is needed, the connector itself and all > 10 existing tasks are restarted, just to make the 11th task run also. My > tasks have a fairly heavy initialization, making it extra annoying. I would > like to see a change, introducing a "mode", where only new/deleted tasks are > started/stopped when notifying the system that the set of tasks changed > (calling context.requestTaskReconfiguration() - or something similar). > Discussed this issue a little on d...@kafka.apache.org in the thread "Kafka > Connect: To much restarting with a SourceConnector with dynamic set of tasks" -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-5505) Connect: Do not restart connector and existing tasks on task-set change
[ https://issues.apache.org/jira/browse/KAFKA-5505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16102200#comment-16102200 ] Per Steffensen commented on KAFKA-5505: --- bq. There's been some discussion about more incremental rebalancing, but as you add/remove tasks, there's no way to avoid the fact that to keep the work balanced we may need to stop/start/move some tasks I can handle restart of tasks! It is just a significant overhead if it happens all the time - and it does, as long a all tasks are restarted every time the set of tasks changes. It will not be a problem if it happens "from time to time" due to rebalance. What annoys me most is actually that the connector itself is restarted, when the set of tasks changes - there is no good reason for that at all, as I see it? The problem is that it takes some time before my connector can builds up its set of tasks, after is (re)starts, because it has to talk with other components to get the entire set of tasks. But the connector has to give a set of tasks almost immediately after (re)start, or things will start behaving strange. Therefore my connector has to start out saying that its set of tasks is empty, and then change the set of tasks (calling context.requestTaskReconfiguration) along the way, as it knows about more and more tasks. But when it does so, the connector is restarted itself, and starts over with an empty set of tasks. I makes the process go: connector started -> empty set of tasks -> some tasks -> connector restarted -> empty set of tasks -> some tasks -> connector restarted -> ... I really have to hack to make it work. If we could just make a change where the connector is not restarted, when it changes its set of tasks, it will be a big step. bq. Can you explain why you have task sets changing so frequently? Ohhh, it is a fairly long explanation in my case. But in general I do not have a hard time imagining connectors with a changing set of tasks. I believe you already have a source-connector out-of-the-box that can copy from a relational database table. Imagine that you would like to extend it, to be able to copy all tables of that database, running one task per table. Guess that would be a fairly reasonable extension. If the set of tables change often, the set of tasks of this connector would change often. bq. It's possible that a different way of assigning partitions to tasks might avoid rebalancing all the time. Well I did that for now. Actually I changed it so that I always have exactly one task, and inside that single task, I handle all the stuff that would otherwise be distributed between tasks. My single task, runs one thread per "partition in the source" - basically one thread where I would like to have had one task. It works the same, but it will not scale, because one task has to run on one machine. Being able to split into several tasks, will help scale. One machine will definitely be able to handle one "partition in the source", but it may not be able to handle "all partitions in the source". I could also take this principle and scale to another fixed number (N) of tasks, higher than one. Then task no M (M from 0 to N-1) will handle "partitions in the source" P where hash(id-of-P) modulo N is M. So I have ways around the problem, but I think the requested change would be nice in general, and something people will expect to be available, especially since it is possible to change the set of tasks along the way - I know I was surprised that it did not already work as I request here. > Connect: Do not restart connector and existing tasks on task-set change > --- > > Key: KAFKA-5505 > URL: https://issues.apache.org/jira/browse/KAFKA-5505 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Affects Versions: 0.10.2.1 >Reporter: Per Steffensen > > I am writing a connector with a frequently changing task-set. It is really > not working very well, because the connector and all existing tasks are > restarted when the set of tasks changes. E.g. if the connector is running > with 10 tasks, and an additional task is needed, the connector itself and all > 10 existing tasks are restarted, just to make the 11th task run also. My > tasks have a fairly heavy initialization, making it extra annoying. I would > like to see a change, introducing a "mode", where only new/deleted tasks are > started/stopped when notifying the system that the set of tasks changed > (calling context.requestTaskReconfiguration() - or something similar). > Discussed this issue a little on d...@kafka.apache.org in the thread "Kafka > Connect: To much restarting with a SourceConnector with dynamic set of tasks" -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-5505) Connect: Do not restart connector and existing tasks on task-set change
[ https://issues.apache.org/jira/browse/KAFKA-5505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16095747#comment-16095747 ] Ewen Cheslack-Postava commented on KAFKA-5505: -- There's been some discussion about more incremental rebalancing, but as you add/remove tasks, there's no way to avoid the fact that to keep the work balanced we may need to stop/start/move some tasks. Can you explain why you have task sets changing so frequently? It's possible that a different way of assigning partitions to tasks might avoid rebalancing all the time. > Connect: Do not restart connector and existing tasks on task-set change > --- > > Key: KAFKA-5505 > URL: https://issues.apache.org/jira/browse/KAFKA-5505 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Affects Versions: 0.10.2.1 >Reporter: Per Steffensen > > I am writing a connector with a frequently changing task-set. It is really > not working very well, because the connector and all existing tasks are > restarted when the set of tasks changes. E.g. if the connector is running > with 10 tasks, and an additional task is needed, the connector itself and all > 10 existing tasks are restarted, just to make the 11th task run also. My > tasks have a fairly heavy initialization, making it extra annoying. I would > like to see a change, introducing a "mode", where only new/deleted tasks are > started/stopped when notifying the system that the set of tasks changed > (calling context.requestTaskReconfiguration() - or something similar). > Discussed this issue a little on d...@kafka.apache.org in the thread "Kafka > Connect: To much restarting with a SourceConnector with dynamic set of tasks" -- This message was sent by Atlassian JIRA (v6.4.14#64029)
