liant (I actually wanted the same thing on the Kafka
> client side, alas
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-795%3A+Add+public+APIs+for+AbstractCoordinator).
> I think it should play well with the future Connect's counterpart of
> KIP-848 (new consumer rebalanc
dd+public+APIs+for+AbstractCoordinator).
> > I think it should play well with the future Connect's counterpart of
> > KIP-848 (new consumer rebalance protocol).
> >
> > I don't want to hijack this thread, but will definitely raise a KIP and
> > start a discussion around
definitely raise a KIP and start
> a discussion around this idea.
>
> From: dev@kafka.apache.org At: 10/20/23 07:21:11 UTC-4:00To:
> dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-987: Connect Static Assignments
>
> Hi Greg,
>
> Thanks for the reply.
>
> I sti
counterpart of KIP-848
(new consumer rebalance protocol).
I don't want to hijack this thread, but will definitely raise a KIP and start a
discussion around this idea.
From: dev@kafka.apache.org At: 10/20/23 07:21:11 UTC-4:00To:
dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-987: Connect Static
Hi Greg,
Thanks for the reply.
I still find the proposed mechanism limited and I'm not sure it really
addressed the pain points I've experienced with Connect.
As you said different tasks from a connector may have different
workload. Connectors may also change the assignment of tasks at
runtime
Hey Sagar,
Thanks for the questions. I hope you find the answers satisfying:
1. This is detailed in the KIP two sentences earlier: "If the
connect.protocol is set to static, each worker will send it's
static.connectors and static.tasks to the coordinator during
rebalances."
2. If you have a
Hi Greg,
Thanks for the KIP. I have a few questions/comments:
1) You mentioned that during a rebalance if all the members of the cluster
support the static protocol, then it would use the steps outlined in the
Proposed Section to do the assignments. In those steps, the leader
identifies the
Hi Mickael,
I'm not Chris but I hope I can still respond to your questions :)
1a. This system behaves best when connectors and tasks are known in
advance, but I don't think that is a requirement. I clarified that the
worker configurations are permissive of non-existent jobs, which
allows for
Hey Tom,
Thanks for your comments! I appreciate your insight here and on the
Strimzi proposal.
1. Fixed, thanks.
2. We haven't removed a protocol yet, so workers still announce that
they can use the old protocol versions, and the group coordinator will
choose the highest mutually supported
Hi Chris,
Thanks for the KIP!
1) With this mechanism it seems Connect administrators have to know in
advanced the names of connectors that will be running. As far as I can
tell, static.connectors and static.tasks will require restarting the
worker to change. Also you don't specify if these
Hi Greg,
Many thanks for the KIP. Here are a few initial questions
1. Incomplete sentence: "But Connect is not situated to be able to manage
resources directly, as workers are given a fixed "
2. You explain how sessioned is now a subset of static, but what happens in
a cluster where some workers
Hey everyone!
I'd like to propose an improvement to Kafka Connect's scheduling
algorithm, with the goal of improving the operability of connect
clusters through resource isolation.
Find the KIP here:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-987%3A+Connect+Static+Assignments
This
12 matches
Mail list logo
