Are the four brokers across two clusters (B1+B2) / (B3+B4), or one cluster? If one cluster, are using the same config/offset/status topics for each Connect cluster? Because that definitely won't work.
In case it's useful: https://rmoff.net/2019/11/22/common-mistakes-made-when-configuring-multiple-kafka-connect-workers/ -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Tue, 17 Mar 2020 at 17:58, mandeep gandhi <welcomemand...@gmail.com> wrote: > Gentle reminder. > > + users. > > On Mon, Mar 16, 2020 at 11:27 PM mandeep gandhi <welcomemand...@gmail.com> > wrote: > > > Hi, > > > > (PS - I did not use users list as this concerns some internals) > > > > I was trying to deploy the following config on a K8s cluster - > > > > Namespace - X1 Connect group id - G1 Bootstrap servers - B1, B2 > > Namespace - X2 Connect group id - G1 Bootstrap servers - B3, B4 > > > > With this configuration, I am seeing multiple rebalances (logs below). So > > the question is how is the Membership protocol working? > > I have tried to follow some KIPs and they mostly say that the Bootstrap > > server is the coordinator of the group. If yes, then logically this > > configuration should work just fine as both configs have different > > bootstrap servers. But as far as I have tried to understand the code > (and > > tried to run Kafka Connect), it seems like the members get added in the > > group one by one and the head of the list becomes the group > co-ordinator. ( > > if I change Connect Group id in 2nd config, things work) > > > > Also, I wanted some pointers on what is happening at the server-side vs > > client-side during the protocol. > > > > > > Kindly help. > > > > Logs - > > > > [2020-03-16 10:27:13,386] INFO [Worker clientId=connect-1, > groupId=streaming-connect] Current config state offset 12 does not match > group assignment 9164. Forcing rebalance. > (org.apache.kafka.connect.runtime.distributed.DistributedHerder:942) > > [2020-03-16 10:27:13,386] INFO [Worker clientId=connect-1, > groupId=streaming-connect] Rebalance started > (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator:213) > > [2020-03-16 10:27:13,386] INFO [Worker clientId=connect-1, > groupId=streaming-connect] (Re-)joining group > (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:505) > > [2020-03-16 10:27:13,388] INFO [Worker clientId=connect-1, > groupId=streaming-connect] Successfully joined group with generation 18535 > (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:469) > > [2020-03-16 10:27:13,389] INFO [Worker clientId=connect-1, > groupId=streaming-connect] Joined group at generation 18535 and got > assignment: Assignment{error=0, > leader='connect-1-3580fdcc-257e-40e7-8243-f7839021599f', leaderUrl=' > http://10.13.191.32:8083/', offset=9164, connectorIds=[], taskIds=[], > revokedConnectorIds=[], revokedTaskIds=[], delay=0} > (org.apache.kafka.connect.runtime.distributed.DistributedHerder:1397) > > [2020-03-16 10:27:13,389] WARN [Worker clientId=connect-1, > groupId=streaming-connect] Catching up to assignment's config offset. > (org.apache.kafka.connect.runtime.distributed.DistributedHerder:909) > > [2020-03-16 10:27:13,389] INFO [Worker clientId=connect-1, > groupId=streaming-connect] Current config state offset 12 is behind group > assignment 9164, reading to end of config log > (org.apache.kafka.connect.runtime.distributed.DistributedHerder:970) > > > > > > > > Thanks, > > > > Mandeep Gandhi > > > > > > > > >
