[
https://issues.apache.org/jira/browse/KAFKA-7835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suraj Fale updated KAFKA-7835:
------------------------------
Description:
We have 6 connectors under distributed Kafka worker. Each of these connectors
has 18-36 tasks. Each time we add new connector all existing connectors
restarts/re-balance.
*Issue:*
* For each new connector we add, previous/existing connector(s) restarts along
with new connector.
* All existing connectors and their tasks goes into re-balance mode.
Assuming that we want to add 3 new connectors in one flow or call, how can we
do that in one HTTP call rather than 3 HTTP POST calls. Because after each HTTP
call we have to wait for re-balance to finish and wait for all tasks to be up
again across all instances.
Is there any way to create all these 3 connectors for same distributed worker
in one HTTP call?
*If not, can we add this as a improvement?*
Something like
{code:java}
[
{
"name": "connector-1",
"config": {
"connector.class": "Connector1Class",
"tasks.max": "36",
"run.next.start": "5",
"fetchsize": "100000",
"topic": "message.connector-1",
"offset.start": "0"
}
},
{
"name": "connector-2",
"config": {
"connector.class": "Connector2Class",
"tasks.max": "18",
"run.next.start": "5",
"fetchsize": "100000",
"topic": "message.connector-2",
"offset.start": "0"
}
},
{
"name": "connector-3",
"config": {
"connector.class": "Connector3Class",
"tasks.max": "18",
"run.next.start": "5",
"fetchsize": "100000",
"topic": "message.connector-3",
"offset.start": "0"
}
}
]
{code}
was:
We have 6 connectors under distributed Kafka worker. Each of these connectors
has 18-36 tasks. Each time we add new connector all existing connectors
restarts/re-balance.
*Issue:*
* For each new connector we add, previous/existing connector(s) restarts along
with new connector.
* All existing connectors and their tasks goes into re-balance mode.
Assuming that we want to add 3 new connectors in one flow or call, how can we
do that in one HTTP call rather than 3 HTTP POST calls. Because after each HTTP
call we have to wait for re-balance to finish and wait for all tasks to be up
again across all instances.
Is there any way to create all these 3 connectors for same distributed worker
in one HTTP call?
*If not, can we add this as a improvement?*
Something like
{code:java}
## http://localhost:8083/connectors
[
{
"name": "connector-1",
"config": {
"connector.class": "Connector1Class",
"tasks.max": "36",
"run.next.start": "5",
"fetchsize": "100000",
"topic": "message.connector-1",
"offset.start": "0"
}
},
{
"name": "connector-2",
"config": {
"connector.class": "Connector2Class",
"tasks.max": "18",
"run.next.start": "5",
"fetchsize": "100000",
"topic": "message.connector-2",
"offset.start": "0"
}
},
{
"name": "connector-3",
"config": {
"connector.class": "Connector3Class",
"tasks.max": "18",
"run.next.start": "5",
"fetchsize": "100000",
"topic": "message.connector-3",
"offset.start": "0"
}
}
]
{code}
> Kafka Connect - Unable to create multiple connectors in a single HTTP (REST)
> call
> ---------------------------------------------------------------------------------
>
> Key: KAFKA-7835
> URL: https://issues.apache.org/jira/browse/KAFKA-7835
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
> Affects Versions: 1.1.1
> Reporter: Suraj Fale
> Priority: Major
> Labels: kafka-connect
>
> We have 6 connectors under distributed Kafka worker. Each of these connectors
> has 18-36 tasks. Each time we add new connector all existing connectors
> restarts/re-balance.
> *Issue:*
> * For each new connector we add, previous/existing connector(s) restarts
> along with new connector.
> * All existing connectors and their tasks goes into re-balance mode.
> Assuming that we want to add 3 new connectors in one flow or call, how can we
> do that in one HTTP call rather than 3 HTTP POST calls. Because after each
> HTTP call we have to wait for re-balance to finish and wait for all tasks to
> be up again across all instances.
> Is there any way to create all these 3 connectors for same distributed worker
> in one HTTP call?
> *If not, can we add this as a improvement?*
> Something like
> {code:java}
> [
> {
> "name": "connector-1",
> "config": {
> "connector.class": "Connector1Class",
> "tasks.max": "36",
> "run.next.start": "5",
> "fetchsize": "100000",
> "topic": "message.connector-1",
> "offset.start": "0"
> }
> },
> {
> "name": "connector-2",
> "config": {
> "connector.class": "Connector2Class",
> "tasks.max": "18",
> "run.next.start": "5",
> "fetchsize": "100000",
> "topic": "message.connector-2",
> "offset.start": "0"
> }
> },
> {
> "name": "connector-3",
> "config": {
> "connector.class": "Connector3Class",
> "tasks.max": "18",
> "run.next.start": "5",
> "fetchsize": "100000",
> "topic": "message.connector-3",
> "offset.start": "0"
> }
> }
> ]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
