Konstantin Lalafaryan created KAFKA-10253:
---------------------------------------------
Summary: Kafka Connect gets into a rebalance loop
Key: KAFKA-10253
URL: https://issues.apache.org/jira/browse/KAFKA-10253
Project: Kafka
Issue Type: Bug
Affects Versions: 2.5.0
Reporter: Konstantin Lalafaryan
Hello everyone!
We are running kafka-connect cluster (3 workers) and very often it gets into a
rebalance loop.
{code:java}
// code placeholder
{code}
2020-07-09 08:51:25,731 INFO [Worker clientId=connect-1, groupId=
kafka-connect] Rebalance started
(org.apache.kafka.connect.runtime.distributed.WorkerCoordinator)
[DistributedHerder-connect-1-1] 2020-07-09 08:51:25,731 INFO [Worker
clientId=connect-1, groupId= kafka-connect] (Re-)joining group
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[DistributedHerder-connect-1-1] 2020-07-09 08:51:25,733 INFO [Worker
clientId=connect-1, groupId= kafka-connect] Was selected to perform
assignments, but do not have latest config found in sync request. Returning an
empty configuration to trigger re-sync.
(org.apache.kafka.connect.runtime.distributed.IncrementalCooperativeAssignor)
[DistributedHerder-connect-1-1] 2020-07-09 08:51:25,735 INFO [Worker
clientId=connect-1, groupId= kafka-connect] Successfully joined group with
generation 305655831
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[DistributedHerder-connect-1-1] 2020-07-09 08:51:25,735 INFO [Worker
clientId=connect-1, groupId= kafka-connect] Joined group at generation
305655831 with protocol version 2 and got assignment: Assignment\{error=1,
leader='connect-1-0008abc5-a152-42fe-a697-a4a4641f72bb',
leaderUrl='http://10.20.30.221:8083/', offset=12, connectorIds=[], taskIds=[],
revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0
(org.apache.kafka.connect.runtime.distributed.DistributedHerder)
[DistributedHerder-connect-1-1] 2020-07-09 08:51:25,735 INFO [Worker
clientId=connect-1, groupId= kafka-connect] Rebalance started
(org.apache.kafka.connect.runtime.distributed.WorkerCoordinator)
[DistributedHerder-connect-1-1] 2020-07-09 08:51:25,735 INFO [Worker
clientId=connect-1, groupId= kafka-connect] (Re-)joining group
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[DistributedHerder-connect-1-1] 2020-07-09 08:51:25,736 INFO [Worker
clientId=connect-1, groupId= kafka-connect] Was selected to perform
assignments, but do not have latest config found in sync request. Returning an
empty configuration to trigger re-sync.
(org.apache.kafka.connect.runtime.distributed.IncrementalCooperativeAssignor)
[DistributedHerder-connect-1-1] 2020-07-09 08:51:25,739 INFO [Worker
clientId=connect-1, groupId= kafka-connect] Successfully joined group with
generation 305655832
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[DistributedHerder-connect-1-1] 2020-07-09 08:51:25,739 INFO [Worker
clientId=connect-1, groupId= kafka-connect] Joined group at generation
305655832 with protocol version 2 and got assignment: Assignment\{error=1,
leader='connect-1-0008abc5-a152-42fe-a697-a4a4641f72bb',
leaderUrl='http://10.20.30.221:8083/', offset=12, connectorIds=[], taskIds=[],
revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0
(org.apache.kafka.connect.runtime.distributed.DistributedHerder)
[DistributedHerder-connect-1-1] 2020-07-09 08:51:25,739 INFO [Worker
clientId=connect-1, groupId= kafka-connect] Rebalance started
(org.apache.kafka.connect.runtime.distributed.WorkerCoordinator)
[DistributedHerder-connect-1-1] 2020-07-09 08:51:25,739 INFO [Worker
clientId=connect-1, groupId= kafka-connect] (Re-)joining group
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[DistributedHerder-connect-1-1] 2020-07-09 08:51:25,740 INFO [Worker
clientId=connect-1, groupId= kafka-connect] Was selected to perform
assignments, but do not have latest config found in sync request. Returning an
empty configuration to trigger re-sync.
(org.apache.kafka.connect.runtime.distributed.IncrementalCooperativeAssignor)
[DistributedHerder-connect-1-1] 2020-07-09 08:51:25,742 INFO [Worker
clientId=connect-1, groupId= kafka-connect] Successfully joined group with
generation 305655833
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[DistributedHerder-connect-1-1] 2020-07-09 08:51:25,742 INFO [Worker
clientId=connect-1, groupId= kafka-connect] Joined group at generation
305655833 with protocol version 2 and got assignment: Assignment\{error=1,
leader='connect-1-0008abc5-a152-42fe-a697-a4a4641f72bb',
leaderUrl='http://10.20.30.221:8083/', offset=12, connectorIds=[], taskIds=[],
revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0
(org.apache.kafka.connect.runtime.distributed.DistributedHerder)
[DistributedHerder-connect-1-1] 2020-07-09 08:51:25,742 INFO [Worker
clientId=connect-1, groupId= kafka-connect] Rebalance started
(org.apache.kafka.connect.runtime.distributed.WorkerCoordinator)
[DistributedHerder-connect-1-1] 2020-07-09 08:51:25,742 INFO [Worker
clientId=connect-1, groupId= kafka-connect] (Re-)joining group
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[DistributedHerder-connect-1-1] 2020-07-09 08:51:25,744 INFO [Worker
clientId=connect-1, groupId= kafka-connect] Was selected to perform
assignments, but do not have latest config found in sync request. Returning an
empty configuration to trigger re-sync.
(org.apache.kafka.connect.runtime.distributed.IncrementalCooperativeAssignor)
[DistributedHerder-connect-1-1] 2020-07-09 08:51:25,746 INFO [Worker
clientId=connect-1, groupId= kafka-connect] Successfully joined group with
generation 305655834
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[DistributedHerder-connect-1-1] 2020-07-09 08:51:25,746 INFO [Worker
clientId=connect-1, groupId= kafka-connect] Joined group at generation
305655834 with protocol version 2 and got assignment: Assignment\{error=1,
leader='connect-1-0008abc5-a152-42fe-a697-a4a4641f72bb',
leaderUrl='http://10.20.30.221:8083/', offset=12, connectorIds=[], taskIds=[],
revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0
(org.apache.kafka.connect.runtime.distributed.DistributedHerder)
[DistributedHerder-connect-1-1] 2020-07-09 08:51:25,746 INFO [Worker
clientId=connect-1, groupId= kafka-connect] Rebalance started
(org.apache.kafka.connect.runtime.distributed.WorkerCoordinator)
[DistributedHerder-connect-1-1] 2020-07-09 08:51:25,746 INFO [Worker
clientId=connect-1, groupId= kafka-connect] (Re-)joining group
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[DistributedHerder-connect-1-1] 2020-07-09 08:51:25,748 INFO [Worker
clientId=connect-1, groupId= kafka-connect] Was selected to perform
assignments, but do not have latest config found in sync request. Returning an
empty configuration to trigger re-sync.
(org.apache.kafka.connect.runtime.distributed.IncrementalCooperativeAssignor)
[DistributedHerder-connect-1-1] 2020-07-09 08:51:25,750 INFO [Worker
clientId=connect-1, groupId= kafka-connect] Successfully joined group with
generation 305655835
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[DistributedHerder-connect-1-1] 2020-07-09 08:51:25,750 INFO [Worker
clientId=connect-1, groupId= kafka-connect] Joined group at generation
305655835 with protocol version 2 and got assignment: Assignment\{error=1,
leader='connect-1-0008abc5-a152-42fe-a697-a4a4641f72bb',
leaderUrl='http://10.20.30.221:8083/', offset=12, connectorIds=[], taskIds=[],
revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0
(org.apache.kafka.connect.runtime.distributed.DistributedHerder)
[DistributedHerder-connect-1-1] 2020-07-09 08:51:25,750 INFO [Worker
clientId=connect-1, groupId= kafka-connect] Rebalance started
(org.apache.kafka.connect.runtime.distributed.WorkerCoordinator)
[DistributedHerder-connect-1-1] 2020-07-09 08:51:25,750 INFO [Worker
clientId=connect-1, groupId= kafka-connect] (Re-)joining group
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[DistributedHerder-connect-1-1] 2020-07-09 08:51:25,751 INFO [Worker
clientId=connect-1, groupId= kafka-connect] Was selected to perform
assignments, but do not have latest config found in sync request. Returning an
empty configuration to trigger re-sync.
(org.apache.kafka.connect.runtime.distributed.IncrementalCooperativeAssignor)
[DistributedHerder-connect-1-1] 2020-07-09 08:51:25,754 INFO [Worker
clientId=connect-1, groupId= kafka-connect] Successfully joined group with
generation 305655836
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[DistributedHerder-connect-1-1] 2020-07-09 08:51:25,754 INFO [Worker
clientId=connect-1, groupId= kafka-connect] Joined group at generation
305655836 with protocol version 2 and got assignment: Assignment\{error=1,
leader='connect-1-0008abc5-a152-42fe-a697-a4a4641f72bb',
leaderUrl='http://10.20.30.221:8083/', offset=12, connectorIds=[], taskIds=[],
revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0
(org.apache.kafka.connect.runtime.distributed.DistributedHerder)
[DistributedHerder-connect-1-1] 2020-07-09 08:51:25,754 INFO [Worker
clientId=connect-1, groupId= kafka-connect] Rebalance started
(org.apache.kafka.connect.runtime.distributed.WorkerCoordinator)
[DistributedHerder-connect-1-1] 2020-07-09 08:51:25,754 INFO [Worker
clientId=connect-1, groupId= kafka-connect] (Re-)joining group
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[DistributedHerder-connect-1-1] 2020-07-09 08:51:25,755 INFO [Worker
clientId=connect-1, groupId= kafka-connect] Was selected to perform
assignments, but do not have latest config found in sync request. Returning an
empty configuration to trigger re-sync.
(org.apache.kafka.connect.runtime.distributed.IncrementalCooperativeAssignor)
[DistributedHerder-connect-1-1] 2020-07-09 08:51:25,757 INFO [Worker
clientId=connect-1, groupId= kafka-connect] Successfully joined group with
generation 305655837
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[DistributedHerder-connect-1-1] 2020-07-09 08:51:25,757 INFO [Worker
clientId=connect-1, groupId= kafka-connect] Joined group at generation
305655837 with protocol version 2 and got assignment: Assignment\{error=1,
leader='connect-1-0008abc5-a152-42fe-a697-a4a4641f72bb',
leaderUrl='http://10.20.30.221:8083/', offset=12, connectorIds=[], taskIds=[],
revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0
(org.apache.kafka.connect.runtime.distributed.DistributedHerder)
[DistributedHerder-connect-1-1] 2020-07-09 08:51:25,757 INFO [Worker
clientId=connect-1, groupId= kafka-connect] Rebalance started
(org.apache.kafka.connect.runtime.distributed.WorkerCoordinator)
[DistributedHerder-connect-1-1] 2020-07-09 08:51:25,757 INFO [Worker
clientId=connect-1, groupId= kafka-connect] (Re-)joining group
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[DistributedHerder-connect-1-1] 2020-07-09 08:51:25,759 INFO [Worker
clientId=connect-1, groupId= kafka-connect] Was selected to perform
assignments, but do not have latest config found in sync request. Returning an
empty configuration to trigger re-sync.
(org.apache.kafka.connect.runtime.distributed.IncrementalCooperativeAssignor)
[DistributedHerder-connect-1-1] 2020-07-09 08:51:25,761 INFO [Worker
clientId=connect-1, groupId= kafka-connect] Successfully joined group with
generation 305655838
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[DistributedHerder-connect-1-1] 2020-07-09 08:51:25,761 INFO [Worker
clientId=connect-1, groupId= kafka-connect] Joined group at generation
305655838 with protocol version 2 and got assignment: Assignment\{error=1,
leader='connect-1-0008abc5-a152-42fe-a697-a4a4641f72bb',
leaderUrl='http://10.20.30.221:8083/', offset=12, connectorIds=[], taskIds=[],
revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0
(org.apache.kafka.connect.runtime.distributed.DistributedHerder)
[DistributedHerder-connect-1-1] 2020-07-09 08:51:25,761 INFO [Worker
clientId=connect-1, groupId= kafka-connect] Rebalance started
(org.apache.kafka.connect.runtime.distributed.WorkerCoordinator)
[DistributedHerder-connect-1-1] 2020-07-09 08:51:25,761 INFO [Worker
clientId=connect-1, groupId= kafka-connect] (Re-)joining group
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[DistributedHerder-connect-1-1] 2020-07-09 08:51:25,763 INFO [Worker
clientId=connect-1, groupId= kafka-connect] Was selected to perform
assignments, but do not have latest config found in sync request. Returning an
empty configuration to trigger re-sync.
(org.apache.kafka.connect.runtime.distributed.IncrementalCooperativeAssignor)
[DistributedHerder-connect-1-1] 2020-07-09 08:51:25,766 INFO [Worker
clientId=connect-1, groupId= kafka-connect] Successfully joined group with
generation 305655839
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[DistributedHerder-connect-1-1] 2020-07-09 08:51:25,766 INFO [Worker
clientId=connect-1, groupId= kafka-connect] Joined group at generation
305655839 with protocol version 2 and got assignment: Assignment\{error=1,
leader='connect-1-0008abc5-a152-42fe-a697-a4a4641f72bb',
leaderUrl='http://10.20.30.221:8083/', offset=12, connectorIds=[], taskIds=[],
revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0
(org.apache.kafka.connect.runtime.distributed.DistributedHerder)
[DistributedHerder-connect-1-1] 2020-07-09 08:51:25,766 INFO [Worker
clientId=connect-1, groupId= kafka-connect] Rebalance started
(org.apache.kafka.connect.runtime.distributed.WorkerCoordinator)
[DistributedHerder-connect-1-1] 2020-07-09 08:51:25,766 INFO [Worker
clientId=connect-1, groupId= kafka-connect] (Re-)joining group
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[DistributedHerder-connect-1-1] 2020-07-09 08:51:25,768 INFO [Worker
clientId=connect-1, groupId= kafka-connect] Was selected to perform
assignments, but do not have latest config found in sync request. Returning an
empty configuration to trigger re-sync.
(org.apache.kafka.connect.runtime.distributed.IncrementalCooperativeAssignor)
[DistributedHerder-connect-1-1] 2020-07-09 08:51:25,771 INFO [Worker
clientId=connect-1, groupId= kafka-connect] Successfully joined group with
generation 305655840
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[DistributedHerder-connect-1-1] 2020-07-09 08:51:25,771 INFO [Worker
clientId=connect-1, groupId= kafka-connect] Joined group at generation
Any feedback is appreciated!
Thanks!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
