[ https://issues.apache.org/jira/browse/KAFKA-7121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shaobo Liu updated KAFKA-7121: ------------------------------ Description: We started a connector, and even though it is in RUNNING status, tasks are not getting assigned: {code:java} {"name":"prod-xxx-v2","connector": {"state":"RUNNING","worker_id":"0.0.0.0:8083"} ,"tasks":[],"type":"sink"} {code} Other connectors are running without issues. Attempt to restart the connector returned 409 status. Logs show the following messages, keep repeating for hours: {code:java} [2018-06-29 20:23:19,288] ERROR Task reconfiguration for prod-xxx-v2 failed unexpectedly, this connector will not be properly reconfigured unless manually triggered. (org.apache.kafka.connect.runtime.distributed.DistributedHerder:956) [2018-06-29 20:23:19,289] INFO 10.200.149.201 - - [29/Jun/2018:20:23:19 +0000] "POST /connectors/prod-xxx-v2/tasks?forward=false HTTP/1.1" 409 113 0 (org.apache.kafka.connect.runtime.rest.RestServer:60) [2018-06-29 20:23:19,289] INFO 10.200.149.201 - - [29/Jun/2018:20:23:19 +0000] "POST /connectors/prod-xxx-v2/tasks?forward=true HTTP/1.1" 409 113 1 (org.apache.kafka.connect.runtime.rest.RestServer:60) [2018-06-29 20:23:19,289] INFO 10.200.149.201 - - [29/Jun/2018:20:23:19 +0000] "POST /connectors/prod-xxx-v2/tasks HTTP/1.1" 409 113 1 (org.apache.kafka.connect.runtime.rest.RestServer:60) [2018-06-29 20:23:19,289] ERROR Request to leader to reconfigure connector tasks failed (org.apache.kafka.connect.runtime.distributed.DistributedHerder:1018) org.apache.kafka.connect.runtime.rest.errors.ConnectRestException: Cannot complete request because of a conflicting operation (e.g. worker rebalance) at org.apache.kafka.connect.runtime.rest.RestServer.httpRequest(RestServer.java:229) at org.apache.kafka.connect.runtime.distributed.DistributedHerder$18.run(DistributedHerder.java:1015) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748){code} was: We started a connector, and even though it is in RUNNING status, tasks are not getting assigned: {"name":"prod-xxx-v2","connector":{"state":"RUNNING","worker_id":"0.0.0.0:8083"},"tasks":[],"type":"sink"} Other connectors are running without issues. Attempt to restart the connector returned 409 status. Logs show the following messages, keep repeating for hours: [2018-06-29 20:23:19,288] ERROR Task reconfiguration for prod-xxx-v2 failed unexpectedly, this connector will not be properly reconfigured unless manually triggered. (org.apache.kafka.connect.runtime.distributed.DistributedHerder:956) [2018-06-29 20:23:19,289] INFO 10.200.149.201 - - [29/Jun/2018:20:23:19 +0000] "POST /connectors/prod-xxx-v2/tasks?forward=false HTTP/1.1" 409 113 0 (org.apache.kafka.connect.runtime.rest.RestServer:60) [2018-06-29 20:23:19,289] INFO 10.200.149.201 - - [29/Jun/2018:20:23:19 +0000] "POST /connectors/prod-xxx-v2/tasks?forward=true HTTP/1.1" 409 113 1 (org.apache.kafka.connect.runtime.rest.RestServer:60) [2018-06-29 20:23:19,289] INFO 10.200.149.201 - - [29/Jun/2018:20:23:19 +0000] "POST /connectors/prod-xxx-v2/tasks HTTP/1.1" 409 113 1 (org.apache.kafka.connect.runtime.rest.RestServer:60) [2018-06-29 20:23:19,289] ERROR Request to leader to reconfigure connector tasks failed (org.apache.kafka.connect.runtime.distributed.DistributedHerder:1018) org.apache.kafka.connect.runtime.rest.errors.ConnectRestException: Cannot complete request because of a conflicting operation (e.g. worker rebalance) at org.apache.kafka.connect.runtime.rest.RestServer.httpRequest(RestServer.java:229) at org.apache.kafka.connect.runtime.distributed.DistributedHerder$18.run(DistributedHerder.java:1015) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) > Intermittently, Connectors fail to assign tasks and keep retrying every > second forever. > --------------------------------------------------------------------------------------- > > Key: KAFKA-7121 > URL: https://issues.apache.org/jira/browse/KAFKA-7121 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect > Reporter: Gwen Shapira > Assignee: Konstantine Karantasis > Priority: Major > > We started a connector, and even though it is in RUNNING status, tasks are > not getting assigned: > > {code:java} > {"name":"prod-xxx-v2","connector": > {"state":"RUNNING","worker_id":"0.0.0.0:8083"} > ,"tasks":[],"type":"sink"} > {code} > > Other connectors are running without issues. > Attempt to restart the connector returned 409 status. > Logs show the following messages, keep repeating for hours: > {code:java} > [2018-06-29 20:23:19,288] ERROR Task reconfiguration for prod-xxx-v2 failed > unexpectedly, this connector will not be properly reconfigured unless > manually triggered. > (org.apache.kafka.connect.runtime.distributed.DistributedHerder:956) > [2018-06-29 20:23:19,289] INFO 10.200.149.201 - - [29/Jun/2018:20:23:19 > +0000] "POST /connectors/prod-xxx-v2/tasks?forward=false HTTP/1.1" 409 113 0 > (org.apache.kafka.connect.runtime.rest.RestServer:60) > [2018-06-29 20:23:19,289] INFO 10.200.149.201 - - [29/Jun/2018:20:23:19 > +0000] "POST /connectors/prod-xxx-v2/tasks?forward=true HTTP/1.1" 409 113 1 > (org.apache.kafka.connect.runtime.rest.RestServer:60) > [2018-06-29 20:23:19,289] INFO 10.200.149.201 - - [29/Jun/2018:20:23:19 > +0000] "POST /connectors/prod-xxx-v2/tasks HTTP/1.1" 409 113 1 > (org.apache.kafka.connect.runtime.rest.RestServer:60) > [2018-06-29 20:23:19,289] ERROR Request to leader to reconfigure connector > tasks failed > (org.apache.kafka.connect.runtime.distributed.DistributedHerder:1018) > org.apache.kafka.connect.runtime.rest.errors.ConnectRestException: Cannot > complete request because of a conflicting operation (e.g. worker rebalance) > at > org.apache.kafka.connect.runtime.rest.RestServer.httpRequest(RestServer.java:229) > at > org.apache.kafka.connect.runtime.distributed.DistributedHerder$18.run(DistributedHerder.java:1015) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748){code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)