Hi Robin
Thanks for your reply.
We are having two worker on different IP. The example which I gave you it
was just a example. We are using kafka version 2.3.1.
Let me tell you again with a simple example.
Suppose, we have two EC2 node, N1 and N2 having worker process W1 and W2
running in distribute mode with groupId i.e in same cluster and two
connectors with having two task each i.e
Node N1: W1 is running
Node N2 : W2 is running
First Connector (C1) : Task1 with id : C1T1 and task 2 with id : C1T2
Second Connector (C2) : Task1 with id : C2T1 and task 2 with id : C2T2
Now Suppose If both W1 and W2 worker process are running and I register
Connector C1 and C2 one after another i.e sequentially, on any of the
worker process, the tasks division between the worker
node are happening like below, which is expected.
*W1* *W2*
C1T1 C1T2
C2T2 C2T2
Now, suppose I stop one worker process e.g W2 and start after some time,
the tasks division is changed like below i.e first connector's task move to
W1 and second connector's task move to W2
*W1* *W2*
C1T1 C2T1
C1T2 C2T2
Please let me know, If it is understandable or not.
Note : Actually, In production, we are gonna have 16 connectors having 10
task each and two worker node. With above scenario, first 8 connectors's
task move to W1 and next 8 connector' task move to W2, Which is not
expected.
Regards and Thanks
Deepak Raghav
On Wed, May 20, 2020 at 1:41 PM Robin Moffatt <ro...@confluent.io> wrote:
> So you're running two workers on the same machine (10.0.0.4), is
> that correct? Normally you'd run one worker per machine unless there was a
> particular reason otherwise.
> What version of Apache Kafka are you using?
> I'm not clear from your question if the distribution of tasks is
> presenting a problem to you (if so please describe why), or if you're just
> interested in the theory behind the rebalancing protocol?
>
>
> --
>
> Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff
>
>
> On Wed, 20 May 2020 at 06:46, Deepak Raghav <deepakragha...@gmail.com>
> wrote:
>
> > Hi
> >
> > Please, can anybody help me with this?
> >
> > Regards and Thanks
> > Deepak Raghav
> >
> >
> >
> > On Tue, May 19, 2020 at 1:37 PM Deepak Raghav <deepakragha...@gmail.com>
> > wrote:
> >
> > > Hi Team
> > >
> > > We have two worker node in a cluster and 2 connector with having 10
> tasks
> > > each.
> > >
> > > Now, suppose if we have two kafka connect process W1(Port 8080) and
> > > W2(Port 8078) started already in distribute mode and then register the
> > > connectors, task of one connector i.e 10 tasks are divided equally
> > between
> > > two worker i.e first task of A connector to W1 worker node and sec task
> > of
> > > A connector to W2 worker node, similarly for first task of B connector,
> > > will go to W1 node and sec task of B connector go to W2 node.
> > >
> > > e.g
> > > *#First Connector : *
> > > {
> > > "name": "REGION_CODE_UPPER-Cdb_Dchchargeableevent",
> > > "connector": {
> > > "state": "RUNNING",
> > > "worker_id": "10.0.0.4:*8080*"
> > > },
> > > "tasks": [
> > > {
> > > "id": 0,
> > > "state": "RUNNING",
> > > "worker_id": "10.0.0.4:*8078*"
> > > },
> > > {
> > > "id": 1,
> > > "state": "RUNNING",
> > > "worker_id": "10.0.0.4:8080"
> > > },
> > > {
> > > "id": 2,
> > > "state": "RUNNING",
> > > "worker_id": "10.0.0.4:8078"
> > > },
> > > {
> > > "id": 3,
> > > "state": "RUNNING",
> > > "worker_id": "10.0.0.4:8080"
> > > },
> > > {
> > > "id": 4,
> > > "state": "RUNNING",
> > > "worker_id": "10.0.0.4:8078"
> > > },
> > > {
> > > "id": 5,
> > > "state": "RUNNING",
> > > "worker_id": "10.0.0.4:8080"
> > > },
> > > {
> > > "id": 6,
> > > "state": "RUNNING",
> > > "worker_id": "10.0.0.4:8078"
> > > },
> > > {
> > > "id": 7,
> > > "state": "RUNNING",
> > > "worker_id": "10.0.0.4:8080"
> > > },
> > > {
> > > "id": 8,
> > > "state": "RUNNING",
> > > "worker_id": "10.0.0.4:8078"
> > > },
> > > {
> > > "id": 9,
> > > "state": "RUNNING",
> > > "worker_id": "10.0.0.4:8080"
> > > }
> > > ],
> > > "type": "sink"
> > > }
> > >
> > >
> > > *#Sec connector*
> > >
> > > {
> > > "name": "REGION_CODE_UPPER-Cdb_Neatransaction",
> > > "connector": {
> > > "state": "RUNNING",
> > > "worker_id": "10.0.0.4:8078"
> > > },
> > > "tasks": [
> > > {
> > > "id": 0,
> > > "state": "RUNNING",
> > > "worker_id": "10.0.0.4:8078"
> > > },
> > > {
> > > "id": 1,
> > > "state": "RUNNING",
> > > "worker_id": "10.0.0.4:8080"
> > > },
> > > {
> > > "id": 2,
> > > "state": "RUNNING",
> > > "worker_id": "10.0.0.4:8078"
> > > },
> > > {
> > > "id": 3,
> > > "state": "RUNNING",
> > > "worker_id": "10.0.0.4:8080"
> > > },
> > > {
> > > "id": 4,
> > > "state": "RUNNING",
> > > "worker_id": "10.0.0.4:8078"
> > > },
> > > {
> > > "id": 5,
> > > "state": "RUNNING",
> > > "worker_id": "10.0.0.4:8080"
> > > },
> > > {
> > > "id": 6,
> > > "state": "RUNNING",
> > > "worker_id": "10.0.0.4:8078"
> > > },
> > > {
> > > "id": 7,
> > > "state": "RUNNING",
> > > "worker_id": "10.0.0.4:8080"
> > > },
> > > {
> > > "id": 8,
> > > "state": "RUNNING",
> > > "worker_id": "10.0.0.4:8078"
> > > },
> > > {
> > > "id": 9,
> > > "state": "RUNNING",
> > > "worker_id": "10.0.0.4:8080"
> > > }
> > > ],
> > > "type": "sink"
> > > }
> > >
> > > But I have seen a strange behavior, when I just shutdown W2 worker node
> > > and start it again task are divided but in diff way i.e all the tasks
> of
> > A
> > > connector will get into W1 node and tasks of B Connector into W2 node.
> > >
> > > Can you please have a look for this.
> > >
> > > *#First Connector*
> > >
> > > {
> > > "name": "REGION_CODE_UPPER-Cdb_Dchchargeableevent",
> > > "connector": {
> > > "state": "RUNNING",
> > > "worker_id": "10.0.0.4:8080"
> > > },
> > > "tasks": [
> > > {
> > > "id": 0,
> > > "state": "RUNNING",
> > > "worker_id": "10.0.0.4:8080"
> > > },
> > > {
> > > "id": 1,
> > > "state": "RUNNING",
> > > "worker_id": "10.0.0.4:8080"
> > > },
> > > {
> > > "id": 2,
> > > "state": "RUNNING",
> > > "worker_id": "10.0.0.4:8080"
> > > },
> > > {
> > > "id": 3,
> > > "state": "RUNNING",
> > > "worker_id": "10.0.0.4:8080"
> > > },
> > > {
> > > "id": 4,
> > > "state": "RUNNING",
> > > "worker_id": "10.0.0.4:8080"
> > > },
> > > {
> > > "id": 5,
> > > "state": "RUNNING",
> > > "worker_id": "10.0.0.4:8080"
> > > },
> > > {
> > > "id": 6,
> > > "state": "RUNNING",
> > > "worker_id": "10.0.0.4:8080"
> > > },
> > > {
> > > "id": 7,
> > > "state": "RUNNING",
> > > "worker_id": "10.0.0.4:8080"
> > > },
> > > {
> > > "id": 8,
> > > "state": "RUNNING",
> > > "worker_id": "10.0.0.4:8080"
> > > },
> > > {
> > > "id": 9,
> > > "state": "RUNNING",
> > > "worker_id": "10.0.0.4:8080"
> > > }
> > > ],
> > > "type": "sink"
> > > }
> > >
> > > *#Second Connector *:
> > >
> > > {
> > > "name": "REGION_CODE_UPPER-Cdb_Neatransaction",
> > > "connector": {
> > > "state": "RUNNING",
> > > "worker_id": "10.0.0.4:8078"
> > > },
> > > "tasks": [
> > > {
> > > "id": 0,
> > > "state": "RUNNING",
> > > "worker_id": "10.0.0.4:8078"
> > > },
> > > {
> > > "id": 1,
> > > "state": "RUNNING",
> > > "worker_id": "10.0.0.4:8078"
> > > },
> > > {
> > > "id": 2,
> > > "state": "RUNNING",
> > > "worker_id": "10.0.0.4:8078"
> > > },
> > > {
> > > "id": 3,
> > > "state": "RUNNING",
> > > "worker_id": "10.0.0.4:8078"
> > > },
> > > {
> > > "id": 4,
> > > "state": "RUNNING",
> > > "worker_id": "10.0.0.4:8078"
> > > },
> > > {
> > > "id": 5,
> > > "state": "RUNNING",
> > > "worker_id": "10.0.0.4:8078"
> > > },
> > > {
> > > "id": 6,
> > > "state": "RUNNING",
> > > "worker_id": "10.0.0.4:8078"
> > > },
> > > {
> > > "id": 7,
> > > "state": "RUNNING",
> > > "worker_id": "10.0.0.4:8078"
> > > },
> > > {
> > > "id": 8,
> > > "state": "RUNNING",
> > > "worker_id": "10.0.0.4:8078"
> > > },
> > > {
> > > "id": 9,
> > > "state": "RUNNING",
> > > "worker_id": "10.0.0.4:8078"
> > > }
> > > ],
> > > "type": "sink"
> > > }
> > >
> > >
> > > Regards and Thanks
> > > Deepak Raghav
> > >
> > >
> >
>
