Hi Robin
Request you to please reply.
Regards and Thanks
Deepak Raghav
On Wed, Jun 10, 2020 at 11:57 AM Deepak Raghav
wrote:
> Hi Robin
>
> Can you please reply.
>
> I just want to add one more thing, that yesterday I tried with
> connect.protocal=eager. Task distribution was balanced after
Hi Robin
Can you please reply.
I just want to add one more thing, that yesterday I tried with
connect.protocal=eager. Task distribution was balanced after that.
Regards and Thanks
Deepak Raghav
On Tue, Jun 9, 2020 at 2:37 PM Deepak Raghav
wrote:
> Hi Robin
>
> Thanks for your reply and acc
Hi Robin
Thanks for your reply and accept my apology for the delayed response.
As you suggested that we should have a separate worker cluster based on
workload pattern. But as you said, task allocation is nondeterministic, so
same things can happen in the new cluster.
Please let me know if my un
The KIP for the current rebalancing protocol is probably a good reference:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-415:+Incremental+Cooperative+Rebalancing+in+Kafka+Connect
--
Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff
On Tue, 26 May 2020 at 14:25, D
Hi Robin
Thanks for the clarification.
As you suggested, that task allocation between the workers is
nondeterministic. I have shared the same information within in my team but
there are some other parties, with whom I need to share this information as
explanation for the issue raised by them and
I don't think you're right to assert that this is "expected behaviour":
> the tasks are divided in below pattern when they are first time
registered
Kafka Connect task allocation is non-determanistic.
I'm still not clear if you're solving for a theoretical problem or an
actual one. If this is a
Hi Robin
I had gone though the link you provided, It is not helpful in my case.
Apart from this, *I am not getting why the tasks are divided in *below
pattern* when they are *first time registered*, which is expected behavior.
I*s there any parameter which we can pass in worker property file which
Thanks for the clarification. If this is an actual problem that you're
encountering and need a solution to then since the task allocation is not
deterministic it sounds like you need to deploy separate worker clusters
based on the workload patterns that you are seeing and machine resources
availabl
Hi Robin
Replying to your query i.e
One thing I'd ask at this point is though if it makes any difference where
the tasks execute?
It actually makes difference to us, we have 16 connectors and as I stated
tasks division earlier, first 8 connector' task are assigned to first
worker process and ano
OK, I understand better now.
You can read more about the guts of the rebalancing protocol that Kafka
Connect uses as of Apache Kafka 2.3 an onwards here:
https://www.confluent.io/blog/incremental-cooperative-rebalancing-in-kafka/
One thing I'd ask at this point is though if it makes any differenc
Hi Robin
Thanks for your reply.
We are having two worker on different IP. The example which I gave you it
was just a example. We are using kafka version 2.3.1.
Let me tell you again with a simple example.
Suppose, we have two EC2 node, N1 and N2 having worker process W1 and W2
running in distri
So you're running two workers on the same machine (10.0.0.4), is
that correct? Normally you'd run one worker per machine unless there was a
particular reason otherwise.
What version of Apache Kafka are you using?
I'm not clear from your question if the distribution of tasks is
presenting a problem
Hi
Please, can anybody help me with this?
Regards and Thanks
Deepak Raghav
On Tue, May 19, 2020 at 1:37 PM Deepak Raghav
wrote:
> Hi Team
>
> We have two worker node in a cluster and 2 connector with having 10 tasks
> each.
>
> Now, suppose if we have two kafka connect process W1(Port 8080)
13 matches
Mail list logo
