Re:[VOTE] KIP-1028: Docker Official Image for Apache Kafka
+1 (non-binding) Thanks Vedarth! From: dev@kafka.apache.org At: 05/14/24 12:13:14 UTC-4:00To: dev@kafka.apache.org Subject: [VOTE] KIP-1028: Docker Official Image for Apache Kafka Hi everyone, I'd like to call a vote on KIP-1028 which aims to introduce a JVM based Docker Official Image (DOI) for Apache Kafka. KIP - https://cwiki.apache.org/confluence/display/KAFKA/KIP-1028%3A+Docker+Official+Im age+for+Apache+Kafka Discussion thread - https://lists.apache.org/thread/6vvwx173jcbgj6vqoq6bo8c0k0ntym0w Thanks and regards, Vedarth
Re:[ANNOUNCE] New Kafka PMC Member: Greg Harris
Congrats! Well deserved From: dev@kafka.apache.org At: 04/13/24 14:42:22 UTC-4:00To: dev@kafka.apache.org Subject: [ANNOUNCE] New Kafka PMC Member: Greg Harris Hi all, Greg Harris has been a Kafka committer since July 2023. He has remained very active and instructive in the community since becoming a committer. It's my pleasure to announce that Greg is now a member of Kafka PMC. Congratulations, Greg! Chris, on behalf of the Apache Kafka PMC
Re:[DISCUSS] KIP-1028: Docker Official Image for Apache Kafka
Hi, What is the difference between this KIP and KIP-975: Docker Image for Apache Kafka? From: dev@kafka.apache.org At: 03/21/24 07:30:07 UTC-4:00To: dev@kafka.apache.org Subject: [DISCUSS] KIP-1028: Docker Official Image for Apache Kafka Hi everyone, I would like to start the discussion on https://cwiki.apache.org/confluence/display/KAFKA/KIP-1028%3A+Docker+Official+Im age+for+Apache+Kafka . This KIP aims to introduce JVM based Docker Official Image (DOI) for Apache Kafka. Regards, Krish.
Re: [VOTE] KIP-1004: Enforce tasks.max property in Kafka Connect
+1 (non-binding) Thanks Chris! From: dev@kafka.apache.org At: 01/02/24 11:49:18 UTC-5:00To: dev@kafka.apache.org Subject: Re: [VOTE] KIP-1004: Enforce tasks.max property in Kafka Connect Hi all, Happy New Year! Wanted to give this a bump now that the holidays are over for a lot of us. Looking forward to people's thoughts! Cheers, Chris On Mon, Dec 4, 2023 at 10:36 AM Chris Egerton wrote: > Hi all, > > I'd like to call for a vote on KIP-1004, which adds enforcement for the > tasks.max connector property in Kafka Connect. > > The KIP: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1004%3A+Enforce+tasks.max+ property+in+Kafka+Connect > > The discussion thread: > https://lists.apache.org/thread/scx75cjwm19jyt19wxky41q9smf5nx6d > > Cheers, > > Chris >
Re:[DISCUSS] KIP-1004: Enforce tasks.max property in Kafka Connect
Thanks for the KIP Chris, adding this check makes total sense. I do have one question. The second paragraph in the Public Interfaces section states: "If the connector generated excessive tasks after being reconfigured, then any existing tasks for the connector will be allowed to continue running, unless that existing set of tasks also exceeds the tasks.max property." Would not failing the connector land us in the second scenario of 'Rejected Alternatives'? From: dev@kafka.apache.org At: 11/11/23 00:27:44 UTC-5:00To: dev@kafka.apache.org Subject: [DISCUSS] KIP-1004: Enforce tasks.max property in Kafka Connect Hi all, I'd like to open up KIP-1004 for discussion: https://cwiki.apache.org/confluence/display/KAFKA/KIP-1004%3A+Enforce+tasks.max+ property+in+Kafka+Connect As a brief summary: this KIP proposes that the Kafka Connect runtime start failing connectors that generate a greater number of tasks than the tasks.max property, with an optional emergency override that can be used to continue running these (probably-buggy) connectors if absolutely necessary. I'll be taking time off most of the next three weeks, so response latency may be a bit higher than usual, but I wanted to kick off the discussion in case we can land this in time for the upcoming 3.7.0 release. Cheers, Chris
Re: [DISCUSS] KIP-987: Connect Static Assignments
Hi, I think different algorithms might work for different workload/scenarios. I have some thoughts that are somewhat tangential to this KIP: it might be a good idea to elevate the ConnectAssignor to the category of plugin, so users can provide their own implementation. The fact that there's a public o.a.k.c.r.distributed.ConnectAssignor interface is brilliant (I actually wanted the same thing on the Kafka client side, alas https://cwiki.apache.org/confluence/display/KAFKA/KIP-795%3A+Add+public+APIs+for+AbstractCoordinator). I think it should play well with the future Connect's counterpart of KIP-848 (new consumer rebalance protocol). I don't want to hijack this thread, but will definitely raise a KIP and start a discussion around this idea. From: dev@kafka.apache.org At: 10/20/23 07:21:11 UTC-4:00To: dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-987: Connect Static Assignments Hi Greg, Thanks for the reply. I still find the proposed mechanism limited and I'm not sure it really addressed the pain points I've experienced with Connect. As you said different tasks from a connector may have different workload. Connectors may also change the assignment of tasks at runtime so for example it task-2 is really busy (because it's assigned a partition with high throughput), this may not be true in 10 minutes as this partition is now assigned to task-1. So having to put which tasks can run on each worker does not really help in this case. I think the "hints" where to place a connector/tasks should come from the connector configuration as it's the engineers building a pipeline that knows best the requirements (in terms of isolation, resources) of their workload. This is basically point 3) in my initial email. The mechanism you propose puts this burden on the cluster administrators who may well not know the workloads and also have to guess/know in advance to properly configure workers. I've not looked into the feasibility but I wonder if a simplified taint/selector approach could give us enough flexibility to make Connect behave better in containerized environments. I understand it's an alternative you rejected but I think could have some benefits. Here is my green field thinking: Add 2 new fields in the connector config: placement and tags. Placement defines the degree of isolation a task requires, it accept 3 values: any (can be placed anywhere like today, the default), colocated (can run on a worker with other tasks from this connector), isolated (requires a dedicated worker). I think these degrees of isolation should cover most use cases. Tags accepts a collections of key=value pair. These can have arbitrary values and are meant to mean something to the management system (for example Strimzi). The accepted values could be configured on the workers by the administrators as they also operate the management system. When a connector is created, the runtime tries to place tasks on the available brokers by matching the placement and tags. If no suitable workers are found, the tasks stay in unassigned state and the runtime waits for the management system to create the necessary workers. We could even envisage to start with only the placement field as in my opinion this is what brings the most value to users. Thanks, Mickael On Wed, Oct 18, 2023 at 8:12 PM Greg Harris wrote: > > Hey Sagar, > > Thanks for the questions. I hope you find the answers satisfying: > > 1. This is detailed in the KIP two sentences earlier: "If the > connect.protocol is set to static, each worker will send it's > static.connectors and static.tasks to the coordinator during > rebalances." > > 2. If you have a static worker and a wildcard worker, the static > worker will be assigned the work preferentially. If the static worker > goes offline, the wildcard worker will be used as a backup. > > 3. I don't think that new Connect users will make use of this feature, > but I've added that clarification. > > 4. Users can implement the strategy you're describing by leaving the > static.connectors field unset. I think that Connect should include > static.connectors for users that do want to control the placement of > connectors. > > 5. Yes. Arbitrary here just means that the assignment is not > influenced by the static assignment. > > 6. Yes. There are no guardrails that ensure that the balance of the > static assignments is better than the builtin algorithm because we > have no method to compare them. > > 7. If the whole cluster uses static assignments with each job only > specified on one worker, the assignments are completely sticky. If a > worker goes offline, those tasks will be offline until that worker > comes back. > If there are multiple workers for a single job, that is specified as > "arbitrary". We could choose to wait for the delay to elapse or > immediately reassign it, the KIP as written could be implemented by > either. > If the assignment would land on a wildcard worker, that should use > cooperative rules, so we
Re: [VOTE] KIP-976: Cluster-wide dynamic log adjustment for Kafka Connect
Good stuff, +1 (non-binding) from me as well De: dev@kafka.apache.org A: 10/09/23 05:16:06 UTC-4:00A: dev@kafka.apache.org Subject: Re: [VOTE] KIP-976: Cluster-wide dynamic log adjustment for Kafka Connect Hi Chris, +1 (non binding) Thanks Fede On Sun, Oct 8, 2023 at 10:11 AM Yash Mayya wrote: > > Hi Chris, > > Thanks for the KIP! > +1 (binding) > > Yash > > On Fri, Oct 6, 2023 at 9:54 PM Greg Harris > wrote: > > > Hey Chris, > > > > Thanks for the KIP! > > I think that preserving the ephemeral nature of the logging change is > > the right choice here, and using the config topic for intra-cluster > > broadcast is better than REST forwarding. > > > > +1 (binding) > > > > Thanks, > > Greg > > > > On Fri, Oct 6, 2023 at 9:05 AM Chris Egerton > > wrote: > > > > > > Hi all, > > > > > > I'd like to call for a vote on KIP-976, which augments the existing > > dynamic > > > logger adjustment REST API for Kafka Connect to apply changes > > cluster-wide > > > instead on a per-worker basis. > > > > > > The KIP: > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-976:+Cluster-wide+dynamic+ log+adjustment+for+Kafka+Connect > > > > > > The discussion thread: > > > https://lists.apache.org/thread/w3x3f3jmyd1vfjxho06y8xgt6mhhzpl5 > > > > > > Cheers, > > > > > > Chris > >
Re: [ANNOUNCE] New committer: Yash Mayya
Congrats! Well deserved From: dev@kafka.apache.org At: 09/21/23 17:05:01 UTC-4:00To: dev@kafka.apache.org Cc: r...@confluent.io.invalid Subject: Re: [ANNOUNCE] New committer: Yash Mayya Congratulations, Yash! On Thu 21. Sep 2023 at 21.57, Randall Hauch wrote: > Congratulations, Yash! > > On Thu, Sep 21, 2023 at 12:31 PM Satish Duggana > wrote: > > > Congratulations Yash!! > > > > On Thu, 21 Sept 2023 at 22:57, Viktor Somogyi-Vass > > wrote: > > > > > > Congrats Yash! > > > > > > On Thu, Sep 21, 2023 at 7:04 PM Josep Prat > > > > wrote: > > > > > > > Congrats Yash! > > > > > > > > ——— > > > > Josep Prat > > > > > > > > Aiven Deutschland GmbH > > > > > > > > Alexanderufer 3-7, 10117 Berlin > > > > > > > > Amtsgericht Charlottenburg, HRB 209739 B > > > > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > > > > > > > m: +491715557497 > > > > > > > > w: aiven.io > > > > > > > > e: josep.p...@aiven.io > > > > > > > > On Thu, Sep 21, 2023, 18:55 Raymond Ng > > wrote: > > > > > > > > > Congrats Yash! Well-deserved! > > > > > > > > > > /Ray > > > > > > > > > > On Thu, Sep 21, 2023 at 9:40 AM Kamal Chandraprakash < > > > > > kamal.chandraprak...@gmail.com> wrote: > > > > > > > > > > > Congratulations Yash! > > > > > > > > > > > > On Thu, Sep 21, 2023, 22:03 Bill Bejeck > > wrote: > > > > > > > > > > > > > Congrats Yash! > > > > > > > > > > > > > > On Thu, Sep 21, 2023 at 12:26 PM Divij Vaidya < > > > > divijvaidy...@gmail.com > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Congratulations Yash! > > > > > > > > > > > > > > > > Divij Vaidya > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 21, 2023 at 6:18 PM Sagar < > > sagarmeansoc...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > > > > > > Congrats Yash ! > > > > > > > > > On Thu, 21 Sep 2023 at 9:38 PM, Ashwin > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Awesome ! Congratulations Yash !! > > > > > > > > > > > > > > > > > > > > On Thu, Sep 21, 2023 at 9:25 PM Edoardo Comar < > > > > > > edoardli...@gmail.com > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Congratulations Yash > > > > > > > > > > > > > > > > > > > > > > On Thu, 21 Sept 2023 at 16:28, Bruno Cadonna < > > > > > cado...@apache.org > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > > > > > > > The PMC of Apache Kafka is pleased to announce a new > > Kafka > > > > > > > > committer > > > > > > > > > > > > Yash Mayya. > > > > > > > > > > > > > > > > > > > > > > > > Yash's major contributions are around Connect. > > > > > > > > > > > > > > > > > > > > > > > > Yash authored the following KIPs: > > > > > > > > > > > > > > > > > > > > > > > > KIP-793: Allow sink connectors to be used with > > > > topic-mutating > > > > > > > SMTs > > > > > > > > > > > > KIP-882: Kafka Connect REST API configuration > > validation > > > > > > timeout > > > > > > > > > > > > improvements > > > > > > > > > > > > KIP-970: Deprecate and remove Connect's redundant > task > > > > > > > > configurations > > > > > > > > > > > > endpoint > > > > > > > > > > > > KIP-980: Allow creating connectors in a stopped state > > > > > > > > > > > > > > > > > > > > > > > > Overall, Yash is known for insightful and friendly > > input to > > > > > > > > discussions > > > > > > > > > > > > and his high quality contributions. > > > > > > > > > > > > > > > > > > > > > > > > Congratulations, Yash! > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > > Bruno (on behalf of the Apache Kafka PMC) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
Re:[VOTE] KIP-970: Deprecate and remove Connect's redundant task configurations endpoint
This makes sense to me, +1 (non-binding) From: dev@kafka.apache.org At: 08/30/23 02:58:59 UTC-4:00To: dev@kafka.apache.org Subject: [VOTE] KIP-970: Deprecate and remove Connect's redundant task configurations endpoint Hi all, This is the vote thread for KIP-970 which proposes deprecating (in the Apache Kafka 3.7 release) and eventually removing (in the next major Apache Kafka release - 4.0) Connect's redundant task configurations endpoint. KIP - https://cwiki.apache.org/confluence/display/KAFKA/KIP-970%3A+Deprecate+and+remov e+Connect%27s+redundant+task+configurations+endpoint Discussion thread - https://lists.apache.org/thread/997qg9oz58kho3c19mdrjodv0n98plvj Thanks, Yash
Re: [VOTE] KIP-953: partition method to be overloaded to accept headers as well.
+1 (non-binding) Thanks for your KIP! From: dev@kafka.apache.org At: 08/16/23 04:48:13 UTC-4:00To: dev@kafka.apache.org Subject: Re: [VOTE] KIP-953: partition method to be overloaded to accept headers as well. Thanks Sagar and Chris for your votes. I will add the details Chris has asked for to the KIP. Hey everyone, Please consider this as a gentle reminder. On Sat, Aug 12, 2023 at 10:48 AM Chris Egerton wrote: > Hi Jack, > > +1 (binding) > > Some friendly, non-blocking suggestions: > > - IMO it's still worth specifying that the headers will be read-only; this > clarifies the intended API contract both for reviewers of the KIP who > haven't read the GitHub PR yet, and for developers who may leverage this > new method > - May be worth mentioning in the compatibility section that any > partitioners that only implement the new interface will be incompatible > with older Kafka clients versions (this is less likely to be a serious > problem in the clients world, but it's a much hairier problem with Connect, > where cross-compatibility between newer/older versions of connectors and > the Kafka Connect runtime is a serious concern) > > Again, these are not blockers and I'm in favor of the KIP with or without > them since I believe both can be addressed at least partially during PR > review and don't have to be tackled at this stage. > > Cheers, > > Chris > > On Sat, Aug 12, 2023 at 12:43 AM Sagar wrote: > > > Hey jack , > > > > +1 (non binding) > > > > Sagar. > > > > On Sat, 12 Aug 2023 at 8:04 AM, Jack Tomy wrote: > > > > > Hey everyone, > > > > > > Please consider this as a gentle reminder. > > > > > > On Mon, Aug 7, 2023 at 5:55 PM Jack Tomy > wrote: > > > > > > > Hey everyone. > > > > > > > > I would like to call for a vote on KIP-953: partition method to be > > > > overloaded to accept headers as well. > > > > > > > > KIP : > > > > > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263424937 > > > > Discussion thread : > > > > https://lists.apache.org/thread/0f20kvfqkmhdqrwcb8vqgqn80szcrcdd > > > > > > > > Thanks > > > > -- > > > > Best Regards > > > > *Jack* > > > > > > > > > > > > > -- > > > Best Regards > > > *Jack* > > > > > > -- Best Regards *Jack*
Re: [VOTE] KIP-959 Add BooleanConverter to Kafka Connect
Thanks Mickael! The KIP has passed with 3 binding votes (Chris Egerton, Greg Harris, Mickael Maison) and 3 non-binding votes (Andrew Schofield, Yash Mayya, Kamal Chandraprakash). I'll update the KIP status, meanwhile the PR is still pending: https://github.com/apache/kafka/pull/14093 From: dev@kafka.apache.org At: 08/08/23 08:33:21 UTC-4:00To: dev@kafka.apache.org Subject: Re: [VOTE] KIP-959 Add BooleanConverter to Kafka Connect Hi, +1 (binding) Thanks for the KIP! On Mon, Aug 7, 2023 at 3:15 PM Hector Geraldino (BLOOMBERG/ 919 3RD A) wrote: > > Hello, > > I still need help from a committer to review/approve this (small) KIP, which adds a new BooleanConverter to the list of converters in Kafka Connect. > > The KIP has a companion PR implementing the feature as well. > > Thanks again! > Sent from Bloomberg Professional for iPhone > > - Original Message - > From: Hector Geraldino > To: dev@kafka.apache.org > At: 08/01/23 11:48:23 UTC-04:00 > > > Hi, > > Still missing one binding vote for this (very small) KIP to pass :) > > From: dev@kafka.apache.org At: 07/28/23 09:37:45 UTC-4:00To: dev@kafka.apache.org > Subject: Re: [VOTE] KIP-959 Add BooleanConverter to Kafka Connect > > Hi everyone, > > Thanks everyone who has reviewed and voted for this KIP. > > So far it has received 3 non-binding votes (Andrew Schofield, Yash Mayya, Kamal > Chandraprakash) and 2 binding votes (Chris Egerton, Greg Harris)- still shy of > one binding vote to pass. > > Can we get help from a committer to push it through? > > Thank you! > Hector > > Sent from Bloomberg Professional for iPhone > > - Original Message - > From: Greg Harris > To: dev@kafka.apache.org > At: 07/26/23 12:23:20 UTC-04:00 > > > Hey Hector, > > Thanks for the straightforward and clear KIP! > +1 (binding) > > Thanks, > Greg > > On Wed, Jul 26, 2023 at 5:16 AM Chris Egerton wrote: > > > > +1 (binding) > > > > Thanks Hector! > > > > On Wed, Jul 26, 2023 at 3:18 AM Kamal Chandraprakash < > > kamal.chandraprak...@gmail.com> wrote: > > > > > +1 (non-binding). Thanks for the KIP! > > > > > > On Tue, Jul 25, 2023 at 11:12 PM Yash Mayya wrote: > > > > > > > Hi Hector, > > > > > > > > Thanks for the KIP! > > > > > > > > +1 (non-binding) > > > > > > > > Thanks, > > > > Yash > > > > > > > > On Tue, Jul 25, 2023 at 11:01 PM Andrew Schofield < > > > > andrew_schofield_j...@outlook.com> wrote: > > > > > > > > > Thanks for the KIP. As you say, not that controversial. > > > > > > > > > > +1 (non-binding) > > > > > > > > > > Thanks, > > > > > Andrew > > > > > > > > > > > On 25 Jul 2023, at 18:22, Hector Geraldino (BLOOMBERG/ 919 3RD A) < > > > > > hgerald...@bloomberg.net> wrote: > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > The changes proposed by KIP-959 (Add BooleanConverter to Kafka > > > Connect) > > > > > have a limited scope and shouldn't be controversial. I'm opening a > > > voting > > > > > thread with the hope that it can be included in the next upcoming 3.6 > > > > > release. > > > > > > > > > > > > Here are some links: > > > > > > > > > > > > KIP: > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-959%3A+Add+BooleanConverte > r+to+Kafka+Connect > > > > > > JIRA: https://issues.apache.org/jira/browse/KAFKA-15248 > > > > > > Discussion thread: > > > > > https://lists.apache.org/thread/15c2t0kl9bozmzjxmkl5n57kv4l4o1dt > > > > > > Pull Request: https://github.com/apache/kafka/pull/14093 > > > > > > > > > > > > Thanks! > > > > > > > > > > > > > > > > > > > > > >
Re: [VOTE] KIP-959 Add BooleanConverter to Kafka Connect
Hello, I still need help from a committer to review/approve this (small) KIP, which adds a new BooleanConverter to the list of converters in Kafka Connect. The KIP has a companion PR implementing the feature as well. Thanks again! Sent from Bloomberg Professional for iPhone - Original Message - From: Hector Geraldino To: dev@kafka.apache.org At: 08/01/23 11:48:23 UTC-04:00 Hi, Still missing one binding vote for this (very small) KIP to pass :) From: dev@kafka.apache.org At: 07/28/23 09:37:45 UTC-4:00To: dev@kafka.apache.org Subject: Re: [VOTE] KIP-959 Add BooleanConverter to Kafka Connect Hi everyone, Thanks everyone who has reviewed and voted for this KIP. So far it has received 3 non-binding votes (Andrew Schofield, Yash Mayya, Kamal Chandraprakash) and 2 binding votes (Chris Egerton, Greg Harris)- still shy of one binding vote to pass. Can we get help from a committer to push it through? Thank you! Hector Sent from Bloomberg Professional for iPhone - Original Message - From: Greg Harris To: dev@kafka.apache.org At: 07/26/23 12:23:20 UTC-04:00 Hey Hector, Thanks for the straightforward and clear KIP! +1 (binding) Thanks, Greg On Wed, Jul 26, 2023 at 5:16 AM Chris Egerton wrote: > > +1 (binding) > > Thanks Hector! > > On Wed, Jul 26, 2023 at 3:18 AM Kamal Chandraprakash < > kamal.chandraprak...@gmail.com> wrote: > > > +1 (non-binding). Thanks for the KIP! > > > > On Tue, Jul 25, 2023 at 11:12 PM Yash Mayya wrote: > > > > > Hi Hector, > > > > > > Thanks for the KIP! > > > > > > +1 (non-binding) > > > > > > Thanks, > > > Yash > > > > > > On Tue, Jul 25, 2023 at 11:01 PM Andrew Schofield < > > > andrew_schofield_j...@outlook.com> wrote: > > > > > > > Thanks for the KIP. As you say, not that controversial. > > > > > > > > +1 (non-binding) > > > > > > > > Thanks, > > > > Andrew > > > > > > > > > On 25 Jul 2023, at 18:22, Hector Geraldino (BLOOMBERG/ 919 3RD A) < > > > > hgerald...@bloomberg.net> wrote: > > > > > > > > > > Hi everyone, > > > > > > > > > > The changes proposed by KIP-959 (Add BooleanConverter to Kafka > > Connect) > > > > have a limited scope and shouldn't be controversial. I'm opening a > > voting > > > > thread with the hope that it can be included in the next upcoming 3.6 > > > > release. > > > > > > > > > > Here are some links: > > > > > > > > > > KIP: > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-959%3A+Add+BooleanConverte r+to+Kafka+Connect > > > > > JIRA: https://issues.apache.org/jira/browse/KAFKA-15248 > > > > > Discussion thread: > > > > https://lists.apache.org/thread/15c2t0kl9bozmzjxmkl5n57kv4l4o1dt > > > > > Pull Request: https://github.com/apache/kafka/pull/14093 > > > > > > > > > > Thanks! > > > > > > > > > > > > > > > > >
Re: [VOTE] KIP-959 Add BooleanConverter to Kafka Connect
Hi, Still missing one binding vote for this (very small) KIP to pass :) From: dev@kafka.apache.org At: 07/28/23 09:37:45 UTC-4:00To: dev@kafka.apache.org Subject: Re: [VOTE] KIP-959 Add BooleanConverter to Kafka Connect Hi everyone, Thanks everyone who has reviewed and voted for this KIP. So far it has received 3 non-binding votes (Andrew Schofield, Yash Mayya, Kamal Chandraprakash) and 2 binding votes (Chris Egerton, Greg Harris)- still shy of one binding vote to pass. Can we get help from a committer to push it through? Thank you! Hector Sent from Bloomberg Professional for iPhone - Original Message - From: Greg Harris To: dev@kafka.apache.org At: 07/26/23 12:23:20 UTC-04:00 Hey Hector, Thanks for the straightforward and clear KIP! +1 (binding) Thanks, Greg On Wed, Jul 26, 2023 at 5:16 AM Chris Egerton wrote: > > +1 (binding) > > Thanks Hector! > > On Wed, Jul 26, 2023 at 3:18 AM Kamal Chandraprakash < > kamal.chandraprak...@gmail.com> wrote: > > > +1 (non-binding). Thanks for the KIP! > > > > On Tue, Jul 25, 2023 at 11:12 PM Yash Mayya wrote: > > > > > Hi Hector, > > > > > > Thanks for the KIP! > > > > > > +1 (non-binding) > > > > > > Thanks, > > > Yash > > > > > > On Tue, Jul 25, 2023 at 11:01 PM Andrew Schofield < > > > andrew_schofield_j...@outlook.com> wrote: > > > > > > > Thanks for the KIP. As you say, not that controversial. > > > > > > > > +1 (non-binding) > > > > > > > > Thanks, > > > > Andrew > > > > > > > > > On 25 Jul 2023, at 18:22, Hector Geraldino (BLOOMBERG/ 919 3RD A) < > > > > hgerald...@bloomberg.net> wrote: > > > > > > > > > > Hi everyone, > > > > > > > > > > The changes proposed by KIP-959 (Add BooleanConverter to Kafka > > Connect) > > > > have a limited scope and shouldn't be controversial. I'm opening a > > voting > > > > thread with the hope that it can be included in the next upcoming 3.6 > > > > release. > > > > > > > > > > Here are some links: > > > > > > > > > > KIP: > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-959%3A+Add+BooleanConverte r+to+Kafka+Connect > > > > > JIRA: https://issues.apache.org/jira/browse/KAFKA-15248 > > > > > Discussion thread: > > > > https://lists.apache.org/thread/15c2t0kl9bozmzjxmkl5n57kv4l4o1dt > > > > > Pull Request: https://github.com/apache/kafka/pull/14093 > > > > > > > > > > Thanks! > > > > > > > > > > > > > > > > >
Re: [VOTE] KIP-959 Add BooleanConverter to Kafka Connect
Hi everyone, Thanks everyone who has reviewed and voted for this KIP. So far it has received 3 non-binding votes (Andrew Schofield, Yash Mayya, Kamal Chandraprakash) and 2 binding votes (Chris Egerton, Greg Harris)- still shy of one binding vote to pass. Can we get help from a committer to push it through? Thank you! Hector Sent from Bloomberg Professional for iPhone - Original Message - From: Greg Harris To: dev@kafka.apache.org At: 07/26/23 12:23:20 UTC-04:00 Hey Hector, Thanks for the straightforward and clear KIP! +1 (binding) Thanks, Greg On Wed, Jul 26, 2023 at 5:16 AM Chris Egerton wrote: > > +1 (binding) > > Thanks Hector! > > On Wed, Jul 26, 2023 at 3:18 AM Kamal Chandraprakash < > kamal.chandraprak...@gmail.com> wrote: > > > +1 (non-binding). Thanks for the KIP! > > > > On Tue, Jul 25, 2023 at 11:12 PM Yash Mayya wrote: > > > > > Hi Hector, > > > > > > Thanks for the KIP! > > > > > > +1 (non-binding) > > > > > > Thanks, > > > Yash > > > > > > On Tue, Jul 25, 2023 at 11:01 PM Andrew Schofield < > > > andrew_schofield_j...@outlook.com> wrote: > > > > > > > Thanks for the KIP. As you say, not that controversial. > > > > > > > > +1 (non-binding) > > > > > > > > Thanks, > > > > Andrew > > > > > > > > > On 25 Jul 2023, at 18:22, Hector Geraldino (BLOOMBERG/ 919 3RD A) < > > > > hgerald...@bloomberg.net> wrote: > > > > > > > > > > Hi everyone, > > > > > > > > > > The changes proposed by KIP-959 (Add BooleanConverter to Kafka > > Connect) > > > > have a limited scope and shouldn't be controversial. I'm opening a > > voting > > > > thread with the hope that it can be included in the next upcoming 3.6 > > > > release. > > > > > > > > > > Here are some links: > > > > > > > > > > KIP: > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-959%3A+Add+BooleanConverter+to+Kafka+Connect > > > > > JIRA: https://issues.apache.org/jira/browse/KAFKA-15248 > > > > > Discussion thread: > > > > https://lists.apache.org/thread/15c2t0kl9bozmzjxmkl5n57kv4l4o1dt > > > > > Pull Request: https://github.com/apache/kafka/pull/14093 > > > > > > > > > > Thanks! > > > > > > > > > > > > > > > > >
Re: Apache Kafka 3.6.0 release
Yes, still need one more binding vote to pass. I'll send a reminder if the vote is still pending after the waiting period. Cheers, From: dev@kafka.apache.org At: 07/26/23 12:17:10 UTC-4:00To: dev@kafka.apache.org Subject: Re: Apache Kafka 3.6.0 release Hi Hector/Yash, Are you planning to reach out to other committers to vote on the KIP and close the vote in the next couple of days? Thanks, Satish. On Wed, 26 Jul 2023 at 20:08, Yash Mayya wrote: > > Hi Hector, > > KIP-959 actually still requires 2 more binding votes to be accepted ( > https://cwiki.apache.org/confluence/display/KAFKA/Bylaws#Bylaws-Approvals). > The non-binding votes from people who aren't committers (including myself) > don't count towards the required lazy majority. > > Thanks, > Yash > > On Wed, Jul 26, 2023 at 7:35 PM Satish Duggana > wrote: > > > Hi Hector, > > Thanks for the update on KIP-959. > > > > ~Satish. > > > > On Wed, 26 Jul 2023 at 18:38, Hector Geraldino (BLOOMBERG/ 919 3RD A) > > wrote: > > > > > > Hi Satish, > > > > > > I added KIP-959 [1] to the list. The KIP has received enough votes to > > pass, but I'm waiting the 72 hours before announcing the results. There's > > also a (small) PR with the implementation for this KIP that hopefully will > > get reviewed/merged soon. > > > > > > Best, > > > > > > [1] > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-959%3A+Add+BooleanConverte r+to+Kafka+Connect > > > > > > From: dev@kafka.apache.org At: 06/12/23 06:22:00 UTC-4:00To: > > dev@kafka.apache.org > > > Subject: Re: Apache Kafka 3.6.0 release > > > > > > Hi, > > > I have created a release plan for Apache Kafka version 3.6.0 on the > > > wiki. You can access the release plan and all related information by > > > following this link: > > > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.6.0 > > > > > > The release plan outlines the key milestones and important dates for > > > version 3.6.0. Currently, the following dates have been set for the > > > release: > > > > > > KIP Freeze: 26th July 23 > > > Feature Freeze : 16th Aug 23 > > > Code Freeze : 30th Aug 23 > > > > > > Please review the release plan and provide any additional information > > > or updates regarding KIPs targeting version 3.6.0. If you have > > > authored any KIPs that are missing a status or if there are incorrect > > > status details, please make the necessary updates and inform me so > > > that I can keep the plan accurate and up to date. > > > > > > Thanks, > > > Satish. > > > > > > On Mon, 17 Apr 2023 at 21:17, Luke Chen wrote: > > > > > > > > Thanks for volunteering! > > > > > > > > +1 > > > > > > > > Luke > > > > > > > > On Mon, Apr 17, 2023 at 2:03 AM Ismael Juma wrote: > > > > > > > > > Thanks for volunteering Satish. +1. > > > > > > > > > > Ismael > > > > > > > > > > On Sun, Apr 16, 2023 at 10:08 AM Satish Duggana < > > satish.dugg...@gmail.com> > > > > > wrote: > > > > > > > > > > > Hi, > > > > > > I would like to volunteer as release manager for the next release, > > > > > > which will be Apache Kafka 3.6.0. > > > > > > > > > > > > If there are no objections, I will start a release plan a week > > after > > > > > > 3.5.0 release(around early May). > > > > > > > > > > > > Thanks, > > > > > > Satish. > > > > > > > > > > > > > > > > > > >
Re: Apache Kafka 3.6.0 release
Sorry, my bad (you can tell this is the first time one of my KIPs have made it this far :)) From: dev@kafka.apache.org At: 07/26/23 10:38:21 UTC-4:00To: dev@kafka.apache.org Subject: Re: Apache Kafka 3.6.0 release Hi Hector, KIP-959 actually still requires 2 more binding votes to be accepted ( https://cwiki.apache.org/confluence/display/KAFKA/Bylaws#Bylaws-Approvals). The non-binding votes from people who aren't committers (including myself) don't count towards the required lazy majority. Thanks, Yash On Wed, Jul 26, 2023 at 7:35 PM Satish Duggana wrote: > Hi Hector, > Thanks for the update on KIP-959. > > ~Satish. > > On Wed, 26 Jul 2023 at 18:38, Hector Geraldino (BLOOMBERG/ 919 3RD A) > wrote: > > > > Hi Satish, > > > > I added KIP-959 [1] to the list. The KIP has received enough votes to > pass, but I'm waiting the 72 hours before announcing the results. There's > also a (small) PR with the implementation for this KIP that hopefully will > get reviewed/merged soon. > > > > Best, > > > > [1] > https://cwiki.apache.org/confluence/display/KAFKA/KIP-959%3A+Add+BooleanConverte r+to+Kafka+Connect > > > > From: dev@kafka.apache.org At: 06/12/23 06:22:00 UTC-4:00To: > dev@kafka.apache.org > > Subject: Re: Apache Kafka 3.6.0 release > > > > Hi, > > I have created a release plan for Apache Kafka version 3.6.0 on the > > wiki. You can access the release plan and all related information by > > following this link: > > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.6.0 > > > > The release plan outlines the key milestones and important dates for > > version 3.6.0. Currently, the following dates have been set for the > > release: > > > > KIP Freeze: 26th July 23 > > Feature Freeze : 16th Aug 23 > > Code Freeze : 30th Aug 23 > > > > Please review the release plan and provide any additional information > > or updates regarding KIPs targeting version 3.6.0. If you have > > authored any KIPs that are missing a status or if there are incorrect > > status details, please make the necessary updates and inform me so > > that I can keep the plan accurate and up to date. > > > > Thanks, > > Satish. > > > > On Mon, 17 Apr 2023 at 21:17, Luke Chen wrote: > > > > > > Thanks for volunteering! > > > > > > +1 > > > > > > Luke > > > > > > On Mon, Apr 17, 2023 at 2:03 AM Ismael Juma wrote: > > > > > > > Thanks for volunteering Satish. +1. > > > > > > > > Ismael > > > > > > > > On Sun, Apr 16, 2023 at 10:08 AM Satish Duggana < > satish.dugg...@gmail.com> > > > > wrote: > > > > > > > > > Hi, > > > > > I would like to volunteer as release manager for the next release, > > > > > which will be Apache Kafka 3.6.0. > > > > > > > > > > If there are no objections, I will start a release plan a week > after > > > > > 3.5.0 release(around early May). > > > > > > > > > > Thanks, > > > > > Satish. > > > > > > > > > > > > > >
Re: Apache Kafka 3.6.0 release
Hi Satish, I added KIP-959 [1] to the list. The KIP has received enough votes to pass, but I'm waiting the 72 hours before announcing the results. There's also a (small) PR with the implementation for this KIP that hopefully will get reviewed/merged soon. Best, [1] https://cwiki.apache.org/confluence/display/KAFKA/KIP-959%3A+Add+BooleanConverter+to+Kafka+Connect From: dev@kafka.apache.org At: 06/12/23 06:22:00 UTC-4:00To: dev@kafka.apache.org Subject: Re: Apache Kafka 3.6.0 release Hi, I have created a release plan for Apache Kafka version 3.6.0 on the wiki. You can access the release plan and all related information by following this link: https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.6.0 The release plan outlines the key milestones and important dates for version 3.6.0. Currently, the following dates have been set for the release: KIP Freeze: 26th July 23 Feature Freeze : 16th Aug 23 Code Freeze : 30th Aug 23 Please review the release plan and provide any additional information or updates regarding KIPs targeting version 3.6.0. If you have authored any KIPs that are missing a status or if there are incorrect status details, please make the necessary updates and inform me so that I can keep the plan accurate and up to date. Thanks, Satish. On Mon, 17 Apr 2023 at 21:17, Luke Chen wrote: > > Thanks for volunteering! > > +1 > > Luke > > On Mon, Apr 17, 2023 at 2:03 AM Ismael Juma wrote: > > > Thanks for volunteering Satish. +1. > > > > Ismael > > > > On Sun, Apr 16, 2023 at 10:08 AM Satish Duggana > > wrote: > > > > > Hi, > > > I would like to volunteer as release manager for the next release, > > > which will be Apache Kafka 3.6.0. > > > > > > If there are no objections, I will start a release plan a week after > > > 3.5.0 release(around early May). > > > > > > Thanks, > > > Satish. > > > > >
[VOTE] KIP-959 Add BooleanConverter to Kafka Connect
Hi everyone, The changes proposed by KIP-959 (Add BooleanConverter to Kafka Connect) have a limited scope and shouldn't be controversial. I'm opening a voting thread with the hope that it can be included in the next upcoming 3.6 release. Here are some links: KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-959%3A+Add+BooleanConverter+to+Kafka+Connect JIRA: https://issues.apache.org/jira/browse/KAFKA-15248 Discussion thread: https://lists.apache.org/thread/15c2t0kl9bozmzjxmkl5n57kv4l4o1dt Pull Request: https://github.com/apache/kafka/pull/14093 Thanks!
Re: [DISCUSS] KIP-959 Add BooleanConverter to Kafka Connect
Thanks Chris for your quick reply. Your suggestions make sense, I amended the KIP and added a note to the class JavaDocs. Also added unit tests to the companion PR [https://github.com/apache/kafka/pull/14093], and will mark it as "Ready for Review" in a few. Cheers From: dev@kafka.apache.org At: 07/25/23 10:42:58 UTC-4:00To: dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-959 Add BooleanConverter to Kafka Connect Hi Hector, Thanks for the KIP! Really appreciate the tight scope, hoping this will be easy to review :) I only have one question: it looks like our existing primitive converters (string converter + subclasses of NumberConverter) are hardcoded to play nicely with null values during deserialization by always providing an optional schema. If that's the intent with this KIP, can we specify that explicitly? (Could be as simple as saying "the schema returned during deserialization will always be an optional boolean schema" with a link to https://kafka.apache.org/35/javadoc/org/apache/kafka/connect/data/Schema.html#OP TIONAL_BOOLEAN_SCHEMA). I don't think we have to say anything else about null handling since FWICT the rest is already handled by the BooleanSerializer and BooleanDeserializer introduced in KIP-907. Cheers, Chris On Tue, Jul 25, 2023 at 9:52 AM Hector Geraldino (BLOOMBERG/ 919 3RD A) < hgerald...@bloomberg.net> wrote: > Hi everyone, > > I'd like to start a discussion of KIP-959, which aims to add a > BooleanConverter to Kafka Connect: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-959%3A+Add+BooleanConverte r+to+Kafka+Connect > > This KIP is a counterpart of KIP-907: Add Boolean Serde to public > interface [ > https://cwiki.apache.org/confluence/display/KAFKA/KIP-907%3A+Add+Boolean+Serde+t o+public+interface], > which added Boolean SerDes to the Kafka serialization APIs. > > The scope of this KIP is very limited, and will help us close a small gap > that exists on the list of included converters for connect's "primitive" > types. > > Looking forward for your feedback. > > Regards, > Hector
[DISCUSS] KIP-959 Add BooleanConverter to Kafka Connect
Hi everyone, I'd like to start a discussion of KIP-959, which aims to add a BooleanConverter to Kafka Connect: https://cwiki.apache.org/confluence/display/KAFKA/KIP-959%3A+Add+BooleanConverter+to+Kafka+Connect This KIP is a counterpart of KIP-907: Add Boolean Serde to public interface [https://cwiki.apache.org/confluence/display/KAFKA/KIP-907%3A+Add+Boolean+Serde+to+public+interface], which added Boolean SerDes to the Kafka serialization APIs. The scope of this KIP is very limited, and will help us close a small gap that exists on the list of included converters for connect's "primitive" types. Looking forward for your feedback. Regards, Hector
Re: [ANNOUNCE] New committer: Greg Harris
Congrats Greg! Well deserved From: dev@kafka.apache.org At: 07/10/23 12:18:48 UTC-4:00To: dev@kafka.apache.org Subject: Re: [ANNOUNCE] New committer: Greg Harris Congratulations! On Mon, Jul 10, 2023 at 9:17 AM Randall Hauch wrote: > > Congratulations, Greg. > > On Mon, Jul 10, 2023 at 11:13 AM Mickael Maison > wrote: > > > Congratulations Greg! > > > > On Mon, Jul 10, 2023 at 6:08 PM Bill Bejeck > > wrote: > > > > > > Congrats Greg! > > > > > > -Bill > > > > > > On Mon, Jul 10, 2023 at 11:53 AM Divij Vaidya > > > wrote: > > > > > > > Congratulations Greg! I am going through a new committer teething > > process > > > > right now and would be happy to get you up to speed. Looking forward to > > > > working with you in your new role. > > > > > > > > -- > > > > Divij Vaidya > > > > > > > > > > > > > > > > On Mon, Jul 10, 2023 at 5:51 PM Josep Prat > > > > > > wrote: > > > > > > > > > Congrats Greg! > > > > > > > > > > > > > > > ——— > > > > > Josep Prat > > > > > > > > > > Aiven Deutschland GmbH > > > > > > > > > > Alexanderufer 3-7, 10117 Berlin > > > > > > > > > > Amtsgericht Charlottenburg, HRB 209739 B > > > > > > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > > > > > > > > > m: +491715557497 > > > > > > > > > > w: aiven.io > > > > > > > > > > e: josep.p...@aiven.io > > > > > > > > > > On Mon, Jul 10, 2023, 17:47 Matthias J. Sax > > wrote: > > > > > > > > > > > Congrats! > > > > > > > > > > > > On 7/10/23 8:45 AM, Chris Egerton wrote: > > > > > > > Hi all, > > > > > > > > > > > > > > The PMC for Apache Kafka has invited Greg Harris to become a > > > > committer, > > > > > > and > > > > > > > we are happy to announce that he has accepted! > > > > > > > > > > > > > > Greg has been contributing to Kafka since 2019. He has made over > > 50 > > > > > > commits > > > > > > > mostly around Kafka Connect and Mirror Maker 2. His most notable > > > > > > > contributions include KIP-898: "Modernize Connect plugin > > discovery" > > > > > and a > > > > > > > deep overhaul of the offset syncing logic in MM2 that addressed > > > > several > > > > > > > technically-difficult, long-standing, high-impact issues. > > > > > > > > > > > > > > He has also been an active participant in discussions and > > reviews on > > > > > the > > > > > > > mailing lists and on GitHub. > > > > > > > > > > > > > > Thanks for all of your contributions, Greg. Congratulations! > > > > > > > > > > > > > > > > > > > > > > > >
Re: [VOTE] KIP-864: Add End-To-End Latency Metrics to Connectors
This will help us greatly. +1 (non-binding) From: dev@kafka.apache.org At: 05/10/23 17:32:35 UTC-4:00To: dev@kafka.apache.org Subject: Re: [VOTE] KIP-864: Add End-To-End Latency Metrics to Connectors Hi everyone, Bumping this vote thread. 2 +1 binding and 1 +1 non-binding so far. Cheers, Jorge. On Mon, 27 Feb 2023 at 18:56, Knowles Atchison Jr wrote: > +1 (non binding) > > On Mon, Feb 27, 2023 at 11:21 AM Chris Egerton > wrote: > > > Hi all, > > > > I could have sworn I +1'd this but I can't seem to find a record of that. > > > > In the hopes that this action is idempotent, +1 (binding). Thanks for the > > KIP! > > > > Cheers, > > > > Chris > > > > On Mon, Feb 27, 2023 at 6:28 AM Mickael Maison > > > wrote: > > > > > Thanks for the KIP > > > > > > +1 (binding) > > > > > > On Thu, Jan 26, 2023 at 4:36 PM Jorge Esteban Quilcate Otoya > > > wrote: > > > > > > > > Hi all, > > > > > > > > I'd like to call for a vote on KIP-864, which proposes to add metrics > > to > > > > measure end-to-end latency in source and sink connectors. > > > > > > > > KIP: > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-864%3A+Add+End-To-End+Late ncy+Metrics+to+Connectors > > > > > > > > Discussion thread: > > > > https://lists.apache.org/thread/k6rh2mr7pg94935fgpqw8b5fj308f2n7 > > > > > > > > Many thanks, > > > > Jorge. > > > > > >
Re:[VOTE] KIP-911: Add source tag to MirrorSourceConnector metrics
+1 (non-binding) From: dev@kafka.apache.org At: 03/15/23 07:08:33 UTC-4:00To: dev@kafka.apache.org Subject: [VOTE] KIP-911: Add source tag to MirrorSourceConnector metrics Hi, I'd like to start the vote on KIP-911 to add the source cluster alias as a tag on the MirrorSourceConnector metrics: https://cwiki.apache.org/confluence/display/KAFKA/KIP-911%3A+Add+source+tag+to+M irrorSourceConnector+metrics Thanks, Mickael
Re:[ANNOUNCE] New Kafka PMC Member: Chris Egerton
Congratulations Chris! Well deserved From: dev@kafka.apache.org At: 03/09/23 13:12:15 UTC-5:00To: dev@kafka.apache.org Subject: [ANNOUNCE] New Kafka PMC Member: Chris Egerton Hi, Everyone, Chris Egerton has been a Kafka committer since July 2022. He has been very instrumental to the community since becoming a committer. It's my pleasure to announce that Chris is now a member of Kafka PMC. Congratulations Chris! Jun on behalf of Apache Kafka PMC
[DISCUSS] KIP-901: Add connectorDeleted flag when stopping tasks
Hi everyone, I've submitted KIP-901, which adds an overloaded Task#stop(boolean connectorDeleted) method to the public Kafka Connect APIs: https://cwiki.apache.org/confluence/display/KAFKA/KIP-901%3A+Add+connectorDeleted+flag+when+stopping+tasks This KIP can be seen as a companion (or counterpart) of KIP-883, which aims to provide the same feature but at the Connector level (there's a separate discussion thread on this list). The KIP also supersedes "KIP-419: Safely notify Kafka Connect SourceTask is stopped". The main goal is to let the task being stopped that it is due to the connector being deleted, so the task can perform any cleanup actions as part of the connector's deletion process. I look forward for your feedback and comments. Thanks! Hector
Re: [DISCUSS] KIP-883: Add delete callback method to Connector API
Thanks for your feedback Chris,
1. I think the behavior should remain the same as it is today. The worker stops
the connector when its configuration is updated, and if the update is a
deletion, it won't start the connector again. If an error happens during stop()
today, the statusListener will update the backing store with a FAILED state.
The only thing that changes on this path is that the Connector#stop() method
will include an additional boolean parameter, so the connector knows that the
reason it is being stopped is because of a deletion, and can perform additional
actions if necessary.
2. I agree; at first I thought it made sense, but after reading KIP-875 and
finding out that connectors can use custom offsets topics to store offsets, I
think this idea needs more refinement. There's probably a way to reuse the work
proposed by this KIP with the "Automatically delete offsets with connectors"
feature mentioned on the "Future work" section of KIP-875, and am happy to
explore it more.
3. I didn't consider that. There is some asymmetry here on how the
StandaloneHerder handles this (tasks are stopped before the connector is) and
the DistributedHerder. One option would be not to handle this on the
#processConnectorConfigUpdates(...) method, but instead wait for the
RebalanceListener#onRevoked(...) callback, which already stops the revoked
connectors and tasks synchronously. The idea would be to enhance this to check
the configState store and, if the configuration of the revoked connector(s) is
gone, then we can let the connector know about that fact when stopping it (by
the aforementioned mechanism). I'll update the KIP and PR if you think it is
worth it.
4. That's correct. As the KIP motivates, we have connectors that need to do
some provisioning/setup when they are deployed (we run connectors for internal
clients), and when tenants delete a connector, we don't have a clear signal
that allows us to cleanup those resources. The goal is probably similar to
https://cwiki.apache.org/confluence/display/KAFKA/KIP-419%3A+Safely+notify+Kafka+Connect+SourceTask+is+stopped,
just took a different approach.
From: dev@kafka.apache.org At: 11/29/22 15:31:31 UTC-5:00To:
dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-883: Add delete callback method to Connector API
Hi Hector,
Thanks for the KIP! Here are my initial thoughts:
1. I like the simplicity of an overloaded stop method, but there is some
asymmetry between stopping a connector and deleting one. If a connector is
stopped (for rebalance, to be reconfigured, etc.) and a failure occurs
then, the failure will be clearly visible in the REST API via, e.g., the
GET /connectors/{connector}/status endpoint. If a connector is deleted and
a failure occurs, with the current proposal, users won't have the same
level of visibility. How can we clearly surface failures caused during the
"destroy" phase of a connector's lifecycle to users?
2. I don't think that this new feature should be used to control (delete)
offsets for connectors. We're addressing that separately in KIP-875, and it
could be a source of headaches for users if they discover that some
connectors' offsets persist across deletion/recreation while others do not.
If anything, we should explicitly recommend against this kind of logic in
the Javadocs for the newly-introduced method.
3. Is it worth trying to give all of the connector's tasks a chance to shut
down before invoking "stop(true)" on the Connector? If so, any thoughts on
how we can accomplish that?
4. Just to make sure we're on the same page--this feature is not being
proposed so that connectors can try to delete the data that they've
produced (i.e., that sink connectors have written to the external system,
or that source connectors have written to Kafka), right?
Cheers,
Chris
On Thu, Nov 17, 2022 at 5:31 PM Hector Geraldino (BLOOMBERG/ 919 3RD A) <
hgerald...@bloomberg.net> wrote:
> Hi,
>
> I've updated the KIP with the new #stop(boolean isDeleted) overloaded
> method, and have also amended the PR and JIRA tickets. I also added a
> couple entries to the "Rejected alternatives" section with the reasons why
> I pivoted from introducing new callback methods to retrofit the existing
> one.
>
> Please let me know what your thoughts are.
>
> Cheers,
> Hector
>
> From: Hector Geraldino (BLOOMBERG/ 919 3RD A) At: 11/16/22 17:38:59
> UTC-5:00To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-883: Add delete callback method to Connector API
>
> Hi Mickael,
>
> I agree that the new STOPPED state proposed in KIP-875 will improve the
> connector lifecycle. The changes proposed in this KIP aim to cover the gap
> where connectors need to actually be deleted, but because the API doesn't
> provide any hooks, external assets are left lingering where they shouldn't.
>
> I agree that this proposal
Re: [DISCUSS] KIP-883: Add delete callback method to Connector API
Hi, I've updated the KIP with the new #stop(boolean isDeleted) overloaded method, and have also amended the PR and JIRA tickets. I also added a couple entries to the "Rejected alternatives" section with the reasons why I pivoted from introducing new callback methods to retrofit the existing one. Please let me know what your thoughts are. Cheers, Hector From: Hector Geraldino (BLOOMBERG/ 919 3RD A) At: 11/16/22 17:38:59 UTC-5:00To: dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-883: Add delete callback method to Connector API Hi Mickael, I agree that the new STOPPED state proposed in KIP-875 will improve the connector lifecycle. The changes proposed in this KIP aim to cover the gap where connectors need to actually be deleted, but because the API doesn't provide any hooks, external assets are left lingering where they shouldn't. I agree that this proposal is similar to KIP-419, maybe the main difference is their focus on Tasks whereas KIP-833 proposes changes to the Connector. My goal is to figure out the correct semantics for notifying connectors that they're being stopped because the connector has been deleted. Now, computing the "deleted" state in both the Standalone and Distributed herders is not hard, so the question is: when shall the connector be notified? The "easiest" option would be to do it by calling an overloaded Connector#stop(deleted) method, but there are other - more expressive - ways, like providing an 'onDelete()' or 'destroy()' method. I'm leaning towards adding an overload method (less complexity, known corner cases), and will amend the KIP with the reasoning behind that decision soon. Thanks for your feedback! From: dev@kafka.apache.org At: 11/16/22 11:13:17 UTC-5:00To: dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-883: Add delete callback method to Connector API Hi Hector, Thanks for the KIP. One tricky aspect is that currently there's no real way to stop a connector so to do so people often just delete them temporarily. KIP-875 proposes adding a mechanism to properly stop connectors which should reduce the need to deleting them and avoid doing potentially expensive cleanup operations repetitively. This KIP also reminds me of KIP-419: https://cwiki.apache.org/confluence/display/KAFKA/KIP-419%3A+Safely+notify+Kafka +Connect+SourceTask+is+stopped. Is it guaranteed the new delete callback will be the last method called? Thanks, Mickael On Tue, Nov 15, 2022 at 5:40 PM Sagar wrote: > > Hey Hector, > > Thanks for the KIP. I have a minor suggestion in terms of naming. Since > this is a callback method, would it make sense to call it onDelete()? > > Also, the failure scenarios discussed by Greg would need handling. Among > other things, I like the idea of having a timeout for graceful shutdown or > else try a force shutdown. What do you think about that approach? > > Thanks! > Sagar. > > On Sat, Nov 12, 2022 at 1:53 AM Hector Geraldino (BLOOMBERG/ 919 3RD A) < > hgerald...@bloomberg.net> wrote: > > > Thanks Greg for taking your time to review not just the KIP but also the > > PR. > > > > 1. You made very valid points regarding the behavior of the destroy() > > callback for connectors that don't follow the happy path. After thinking > > about it, I decided to tweak the implementation a bit and have the > > destroy() method be called during the worker shutdown: this means it will > > share the same guarantees the connector#stop() method has. An alternative > > implementation can be to have an overloaded connector#stop(boolean deleted) > > method that signals a connector that it is being stopped due to deletion, > > but I think that having a separate destroy() method provides clearer > > semantics. > > > > I'll make sure to ammend the KIP with these details. > > > > 3. Without going too deep on the types of operations that can be performed > > by a connector when it's being deleted, I can imagine the > > org.apache.kafka.connect.source.SourceConnector base class having a default > > implementation that deletes the connector's offsets automatically > > (controlled by a property); this is in the context of KIP-875 (first-class > > offsets support in Kafka Connect). Similar behaviors can be introduced for > > the SinkConnector, however I'm not sure if this KIP is the right place to > > discuss all the possibilities, or if we shoold keeping it more > > narrow-focused on providing a callback mechanism for when connectors are > > deleted, and what the expectations are around this newly introduced method. > > What do you think? > > > > > > From: dev@kafka.apache.org At: 11/09/22 16:55:04 UTC-5:00To: > > dev@kafka.apache.org > > Subject: Re: [DISCUSS] KIP-883: Add delete callback method to Co
Re: [DISCUSS] KIP-883: Add delete callback method to Connector API
Hi Mickael, I agree that the new STOPPED state proposed in KIP-875 will improve the connector lifecycle. The changes proposed in this KIP aim to cover the gap where connectors need to actually be deleted, but because the API doesn't provide any hooks, external assets are left lingering where they shouldn't. I agree that this proposal is similar to KIP-419, maybe the main difference is their focus on Tasks whereas KIP-833 proposes changes to the Connector. My goal is to figure out the correct semantics for notifying connectors that they're being stopped because the connector has been deleted. Now, computing the "deleted" state in both the Standalone and Distributed herders is not hard, so the question is: when shall the connector be notified? The "easiest" option would be to do it by calling an overloaded Connector#stop(deleted) method, but there are other - more expressive - ways, like providing an 'onDelete()' or 'destroy()' method. I'm leaning towards adding an overload method (less complexity, known corner cases), and will amend the KIP with the reasoning behind that decision soon. Thanks for your feedback! From: dev@kafka.apache.org At: 11/16/22 11:13:17 UTC-5:00To: dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-883: Add delete callback method to Connector API Hi Hector, Thanks for the KIP. One tricky aspect is that currently there's no real way to stop a connector so to do so people often just delete them temporarily. KIP-875 proposes adding a mechanism to properly stop connectors which should reduce the need to deleting them and avoid doing potentially expensive cleanup operations repetitively. This KIP also reminds me of KIP-419: https://cwiki.apache.org/confluence/display/KAFKA/KIP-419%3A+Safely+notify+Kafka +Connect+SourceTask+is+stopped. Is it guaranteed the new delete callback will be the last method called? Thanks, Mickael On Tue, Nov 15, 2022 at 5:40 PM Sagar wrote: > > Hey Hector, > > Thanks for the KIP. I have a minor suggestion in terms of naming. Since > this is a callback method, would it make sense to call it onDelete()? > > Also, the failure scenarios discussed by Greg would need handling. Among > other things, I like the idea of having a timeout for graceful shutdown or > else try a force shutdown. What do you think about that approach? > > Thanks! > Sagar. > > On Sat, Nov 12, 2022 at 1:53 AM Hector Geraldino (BLOOMBERG/ 919 3RD A) < > hgerald...@bloomberg.net> wrote: > > > Thanks Greg for taking your time to review not just the KIP but also the > > PR. > > > > 1. You made very valid points regarding the behavior of the destroy() > > callback for connectors that don't follow the happy path. After thinking > > about it, I decided to tweak the implementation a bit and have the > > destroy() method be called during the worker shutdown: this means it will > > share the same guarantees the connector#stop() method has. An alternative > > implementation can be to have an overloaded connector#stop(boolean deleted) > > method that signals a connector that it is being stopped due to deletion, > > but I think that having a separate destroy() method provides clearer > > semantics. > > > > I'll make sure to ammend the KIP with these details. > > > > 3. Without going too deep on the types of operations that can be performed > > by a connector when it's being deleted, I can imagine the > > org.apache.kafka.connect.source.SourceConnector base class having a default > > implementation that deletes the connector's offsets automatically > > (controlled by a property); this is in the context of KIP-875 (first-class > > offsets support in Kafka Connect). Similar behaviors can be introduced for > > the SinkConnector, however I'm not sure if this KIP is the right place to > > discuss all the possibilities, or if we shoold keeping it more > > narrow-focused on providing a callback mechanism for when connectors are > > deleted, and what the expectations are around this newly introduced method. > > What do you think? > > > > > > From: dev@kafka.apache.org At: 11/09/22 16:55:04 UTC-5:00To: > > dev@kafka.apache.org > > Subject: Re: [DISCUSS] KIP-883: Add delete callback method to Connector API > > > > Hi Hector, > > > > Thanks for the KIP! > > > > This is certainly missing functionality from the native Connect framework, > > and we should try to make it possible to inform connectors about this part > > of their lifecycle. > > However, as with most functionality that was left out of the initial > > implementation of the framework, the details are more challenging to work > > out. > > > > 1. What happens when the destroy call throws an error, how d
Re: [DISCUSS] KIP-883: Add delete callback method to Connector API
Hi Sagar, Thanks for your feedback! I actually renamed the method from "deleted()" to "destroyed()", which I think conveys the intention more clearly. I can certainly rename it to be 'onDeleted()', although I feel any method named onXXX() belongs to a listener class :) Regarding failure scenarios, an option I'm considering is to just provide an overloaded Connector#stop(boolean deleted) method that is called during WorkerConnector#doShutdown(). This has the advantage of providing the same semantics that the current Connector#stop() has, with the caveat that the API won't be as expressive. Also, the extra 'cleanup' bits that were supposed to happen when a connector is deleted might not to happen at all if the connector doesn't stop before the configured timeout (and is therefore cancelled). At this point I think the simplest option would be to provide an overloaded method (with a default implementation) that connectors can override. Wdyt? From: dev@kafka.apache.org At: 11/15/22 11:40:26 UTC-5:00To: dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-883: Add delete callback method to Connector API Hey Hector, Thanks for the KIP. I have a minor suggestion in terms of naming. Since this is a callback method, would it make sense to call it onDelete()? Also, the failure scenarios discussed by Greg would need handling. Among other things, I like the idea of having a timeout for graceful shutdown or else try a force shutdown. What do you think about that approach? Thanks! Sagar. On Sat, Nov 12, 2022 at 1:53 AM Hector Geraldino (BLOOMBERG/ 919 3RD A) < hgerald...@bloomberg.net> wrote: > Thanks Greg for taking your time to review not just the KIP but also the > PR. > > 1. You made very valid points regarding the behavior of the destroy() > callback for connectors that don't follow the happy path. After thinking > about it, I decided to tweak the implementation a bit and have the > destroy() method be called during the worker shutdown: this means it will > share the same guarantees the connector#stop() method has. An alternative > implementation can be to have an overloaded connector#stop(boolean deleted) > method that signals a connector that it is being stopped due to deletion, > but I think that having a separate destroy() method provides clearer > semantics. > > I'll make sure to ammend the KIP with these details. > > 3. Without going too deep on the types of operations that can be performed > by a connector when it's being deleted, I can imagine the > org.apache.kafka.connect.source.SourceConnector base class having a default > implementation that deletes the connector's offsets automatically > (controlled by a property); this is in the context of KIP-875 (first-class > offsets support in Kafka Connect). Similar behaviors can be introduced for > the SinkConnector, however I'm not sure if this KIP is the right place to > discuss all the possibilities, or if we shoold keeping it more > narrow-focused on providing a callback mechanism for when connectors are > deleted, and what the expectations are around this newly introduced method. > What do you think? > > > From: dev@kafka.apache.org At: 11/09/22 16:55:04 UTC-5:00To: > dev@kafka.apache.org > Subject: Re: [DISCUSS] KIP-883: Add delete callback method to Connector API > > Hi Hector, > > Thanks for the KIP! > > This is certainly missing functionality from the native Connect framework, > and we should try to make it possible to inform connectors about this part > of their lifecycle. > However, as with most functionality that was left out of the initial > implementation of the framework, the details are more challenging to work > out. > > 1. What happens when the destroy call throws an error, how does the > framework respond? > > This is unspecified in the KIP, and it appears that your proposed changes > could cause the herder to fail. > From the perspective of operators & connector developers, what is a > reasonable expectation to have for failure of a destroy? > I could see operators wanting both a graceful-delete to make use of this > new feature, and a force-delete for when the graceful-delete fails. > A connector developer could choose to swallow all errors encountered, or > fail-fast to indicate to the operator that there is an issue with the > graceful-delete flow. > If the alternative is crashing the herder, connector developers may choose > to hide serious errors, which is undesirable. > > 2. What happens when the destroy() call takes a long time to complete, or > is interrupted? > > It appears that your implementation serially destroy()s each appropriate > connector, and may prevent the herder thread from making progress while the > operation is ongoing. > We have previously had to patch Connect to perform al
Re: [DISCUSS] KIP-883: Add delete callback method to Connector API
and connector developers that wish to clean up data/metadata during destroy()? 4. How should connector implementations of destroy handle backwards compatibility? In terms of backward-compatibility for the framework vs connector versions, I think the default-noop method is very reasonable. However, what happens when someone upgrades from a version of a connector without a destroy() implementation to one with an implementation, and maintain backwards compatibility? To replicate the same behavior, the connector might include something like an `enable.cleanup` config which allows users to opt-in to the new behavior. This could mean the proliferation of many different configurations to handle this behavior. Maybe we can provide some recommendations to developers, or some mechanism to standardize this opt-in behavior. I'm interested to hear if you have any experience with the above, if you've experimented with this feature in your fork. Thanks, Greg On Thu, Nov 3, 2022 at 11:55 AM Hector Geraldino (BLOOMBERG/ 919 3RD A) < hgerald...@bloomberg.net> wrote: > Hi everyone, > > I've submitted KIP-883, which introduces a callback to the public > Connector API called when deleting a connector: > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-883%3A+Add+delete+callback +method+to+Connector+API > > It adds a new `deleted()` method (open to better naming suggestions) to > the org.apache.kafka.connect.connector.Connector abstract class, which will > be invoked by connect Workers when a connector is being deleted. > > Feedback and comments are welcome. > > Thank you! > Hector > >
[DISCUSS] KIP-883: Add delete callback method to Connector API
Hi everyone, I've submitted KIP-883, which introduces a callback to the public Connector API called when deleting a connector: https://cwiki.apache.org/confluence/display/KAFKA/KIP-883%3A+Add+delete+callback+method+to+Connector+API It adds a new `deleted()` method (open to better naming suggestions) to the org.apache.kafka.connect.connector.Connector abstract class, which will be invoked by connect Workers when a connector is being deleted. Feedback and comments are welcome. Thank you! Hector
Re:[DISCUSS] KIP-874: TopicRoundRobinAssignor
Hi Mathieu. I took a look at your KIP and have a couple questions. If the goal is to do the partition assignments at a topic level, wouldn't having single-partition topics solve this problem? You also mentioned that your goal is to minimize the potential of a poison pill message breaking all members of a group (by keeping track of which topics have 'failed'), but it is not clear how this can be achieved with this assignor. If we imagine an scenario where: * A group has 3 members (A, B, C) * Members are subscribed to 3 topics (T1, T2, T3) * Each member is assigned one topic (A[T1], B[T2], C[T3]) * One member fails to consume from a topic/partition (B[T2]), and goes into failed state How will the group leader know that T2 should not be re-assigned on the next rebalance? Can you elaborate a bit more on the mechanisms used to communicate this state to the other group members? Thanks From: dev@kafka.apache.org At: 10/05/22 03:47:33 UTC-4:00To: dev@kafka.apache.org Subject: [DISCUSS] KIP-874: TopicRoundRobinAssignor Hi Kafka Developers, My proposal is to add a new partition assignment strategy at the topic level to : - have a better data consistency by consumed topic in case of exception - have a solution much thread safe for the consumer In case there are multiple consumers and multiple topics. Here is the link to the KIP with all the explanations : https://cwiki.apache.org/confluence/x/XozGDQ Thank you in advance for your feedbacks, Mathieu
Connector API callbacks for create/delete events
Hi, We've some custom connectors that require provisioning external resources (think of creating queues, S3 buckets, or activating accounts) when the connector instance is created, but also need to cleanup these resources (delete, deactivate) when the connector instance is deleted. The connector API (org.apache.kafka.connect.connector.Connector) provides a start() and stop() methods and, while we can probably work around the start() method to check if the initialization of external resources has been done, there is currently no hook that a connector can use to perform any cleanup task when it is deleted. I'm planning to write a KIP that enhances the Connector API by having methods that are invoked by the Herder when connectors are created and/or deleted; but before doing so, I wanted to ask the community if there's already some workaround(s) that we can be used to achieve these tasks. Thank you!
Re: [DISCUSS] KIP-848: The Next Generation of the Consumer Rebalance Protocol
Yes, I agree that leadership could be modeled after partition assignment(s). However, I can think of some expanded versions of the 'leader election' use case that exist today in Schema Registry. The advantage of creating a different 'type' that isn't necessarily tied to topics/partitions (and is used only for resource management) would be that we can scale the number of resources it handles (akin to a connect cluster increasing the number of connectors/tasks) without having to change topics/partitions, as these partitions will never have any data (and can't be shrunk), they will be used just for leadership. This is in the spirit of KIP-795. We can table this discussion for after the Connect discussion, as there will be more clarity on how extending the new protocol will look like. From: dev@kafka.apache.org At: 09/12/22 07:58:32 UTC-4:00To: dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-848: The Next Generation of the Consumer Rebalance Protocol Hi Hector, We definitely need to share internals with Connect APIs. That model would not scale otherwise. Regarding the schema registry, I wonder if we could just use the new protocol. At the end of the day, the schema registry wants to elect a single writer for a partition and the owner of the partition can be considered as the leader. I haven't really tried this out but that seems doable. What do you think? Best, David On Fri, Sep 9, 2022 at 8:45 PM Hector Geraldino (BLOOMBERG/ 919 3RD A) wrote: > > So it seems there's a consensus on having dedicated APIs for Connect, which means having data model (group, member, assignment) and APIs (heartbeat request/response, assignment prepare and install) tailored specifically to connect. I wonder if adding support for other coordinator group types (e.g. leadership, in the case of schema registry) will require similar assets (new model classes to express members and resources, custom heartbeats and assignment prepare/install APIs). > > I think that, as new use cases are considered, the core primitives of the new protocol will be generalized, so new types don't have to implement the whole stack (e.g. state machines), but only functions like detecting group metadata changes, or computing assignments of the resources handled by each type (Topic/Partitions in the case of consumer, Connector/Task in the case of Connect, Leadership in the case of Schema Registry, and so on). > > > From: dev@kafka.apache.org At: 08/12/22 09:31:36 UTC-4:00To: dev@kafka.apache.org > Subject: Re: [DISCUSS] KIP-848: The Next Generation of the Consumer Rebalance Protocol > > Thank you Guozhang/David for the feedback. Looks like there's agreement on > using separate APIs for Connect. I would revisit the doc and see what > changes are to be made. > > Thanks! > Sagar. > > On Tue, Aug 9, 2022 at 7:11 PM David Jacot > wrote: > > > Hi Sagar, > > > > Thanks for the feedback and the document. That's really helpful. I > > will take a look at it. > > > > Overall, it seems to me that both Connect and the Consumer could share > > the same underlying "engine". The main difference is that the Consumer > > assigns topic-partitions to members whereas Connect assigns tasks to > > workers. I see two ways to move forward: > > 1) We extend the new proposed APIs to support different resource types > > (e.g. partitions, tasks, etc.); or > > 2) We use new dedicated APIs for Connect. The dedicated APIs would be > > similar to the new ones but different on the content/resources and > > they would rely on the same engine on the coordinator side. > > > > I personally lean towards 2) because I am not a fan of overcharging > > APIs to serve different purposes. That being said, I am not opposed to > > 1) if we can find an elegant way to do it. > > > > I think that we can continue to discuss it here for now in order to > > ensure that this KIP is compatible with what we will do for Connect in > > the future. > > > > Best, > > David > > > > On Mon, Aug 8, 2022 at 2:41 PM David Jacot wrote: > > > > > > Hi all, > > > > > > I am back from vacation. I will go through and address your comments > > > in the coming days. Thanks for your feedback. > > > > > > Cheers, > > > David > > > > > > On Wed, Aug 3, 2022 at 10:05 PM Gregory Harris > > wrote: > > > > > > > > Hey All! > > > > > > > > Thanks for the KIP, it's wonderful to see cooperative rebalancing > > making it > > > > down the stack! > > > > > > > > I had a few questions: > > > > > > > > 1. The 'Rejected Alternatives' section describes how member epoch >
Re:[VOTE] KIP-848: The Next Generation of the Consumer Rebalance Protocol
+1 (non-binding) Really looking forward for the discussion on how other group types (especially connect) will support this new protocol. From: dev@kafka.apache.org At: 09/09/22 04:32:46 UTC-4:00To: dev@kafka.apache.org Subject: [VOTE] KIP-848: The Next Generation of the Consumer Rebalance Protocol Hi all, Thank you all for the very positive discussion about KIP-848. It looks like folks are very positive about it overall. I would like to start a vote on KIP-848, which introduces a brand new consumer rebalance protocol. The KIP is here: https://cwiki.apache.org/confluence/x/HhD1D. Best, David
Re: [DISCUSS] KIP-848: The Next Generation of the Consumer Rebalance Protocol
So it seems there's a consensus on having dedicated APIs for Connect, which means having data model (group, member, assignment) and APIs (heartbeat request/response, assignment prepare and install) tailored specifically to connect. I wonder if adding support for other coordinator group types (e.g. leadership, in the case of schema registry) will require similar assets (new model classes to express members and resources, custom heartbeats and assignment prepare/install APIs). I think that, as new use cases are considered, the core primitives of the new protocol will be generalized, so new types don't have to implement the whole stack (e.g. state machines), but only functions like detecting group metadata changes, or computing assignments of the resources handled by each type (Topic/Partitions in the case of consumer, Connector/Task in the case of Connect, Leadership in the case of Schema Registry, and so on). From: dev@kafka.apache.org At: 08/12/22 09:31:36 UTC-4:00To: dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-848: The Next Generation of the Consumer Rebalance Protocol Thank you Guozhang/David for the feedback. Looks like there's agreement on using separate APIs for Connect. I would revisit the doc and see what changes are to be made. Thanks! Sagar. On Tue, Aug 9, 2022 at 7:11 PM David Jacot wrote: > Hi Sagar, > > Thanks for the feedback and the document. That's really helpful. I > will take a look at it. > > Overall, it seems to me that both Connect and the Consumer could share > the same underlying "engine". The main difference is that the Consumer > assigns topic-partitions to members whereas Connect assigns tasks to > workers. I see two ways to move forward: > 1) We extend the new proposed APIs to support different resource types > (e.g. partitions, tasks, etc.); or > 2) We use new dedicated APIs for Connect. The dedicated APIs would be > similar to the new ones but different on the content/resources and > they would rely on the same engine on the coordinator side. > > I personally lean towards 2) because I am not a fan of overcharging > APIs to serve different purposes. That being said, I am not opposed to > 1) if we can find an elegant way to do it. > > I think that we can continue to discuss it here for now in order to > ensure that this KIP is compatible with what we will do for Connect in > the future. > > Best, > David > > On Mon, Aug 8, 2022 at 2:41 PM David Jacot wrote: > > > > Hi all, > > > > I am back from vacation. I will go through and address your comments > > in the coming days. Thanks for your feedback. > > > > Cheers, > > David > > > > On Wed, Aug 3, 2022 at 10:05 PM Gregory Harris > wrote: > > > > > > Hey All! > > > > > > Thanks for the KIP, it's wonderful to see cooperative rebalancing > making it > > > down the stack! > > > > > > I had a few questions: > > > > > > 1. The 'Rejected Alternatives' section describes how member epoch > should > > > advance in step with the group epoch and assignment epoch values. I > think > > > that this is a good idea for the reasons described in the KIP. When the > > > protocol is incrementally assigning partitions to a worker, what member > > > epoch does each incremental assignment use? Are member epochs re-used, > and > > > a single member epoch can correspond to multiple different > (monotonically > > > larger) assignments? > > > > > > 2. Is the Assignor's 'Reason' field opaque to the group coordinator? If > > > not, should custom client-side assignor implementations interact with > the > > > Reason field, and how is its common meaning agreed upon? If so, what > is the > > > benefit of a distinct Reason field over including such functionality > in the > > > opaque metadata? > > > > > > 3. The following is included in the KIP: "Thanks to this, the input of > the > > > client side assignor is entirely driven by the group coordinator. The > > > consumer is no longer responsible for maintaining any state besides its > > > assigned partitions." Does this mean that the client-side assignor MAY > > > incorporate additional non-Metadata state (such as partition > throughput, > > > cpu/memory metrics, config topics, etc), or that additional > non-Metadata > > > state SHOULD NOT be used? > > > > > > 4. I see that there are separate classes > > > for org.apache.kafka.server.group.consumer.PartitionAssignor > > > and org.apache.kafka.clients.consumer.PartitionAssignor that seem to > > > overlap significantly. Is it possible for these two implementations to > be > > > unified? This would serve to promote feature parity of server-side and > > > client-side assignors, and would also facilitate operational > flexibility in > > > certain situations. For example, if a server-side assignor has some > poor > > > behavior and needs a patch, deploying the patched assignor to the > client > > > and switching one consumer group to a client-side assignor may be > faster > > > and less risky than patching all of the brokers. With the currently
Re: Consumer Lag-Apache_kafka_JMX metrics
As far as I know, such metric does not exist. Strictly speaking, consumer lag can be defined as the difference between the last produced offset (high watermark) and the last committed offset by the group, but such metric has very little value without considering the time dimension. It'd be tricky for the broker to report on consumer 'lag', as the concept of lag itself varies. You already know about Burrow (and I recall reading about Uber's uGroup), and you already see that it considers a consumer lagging if it is not making enough progress in a sliding time window (10 mins?). But other tools/use cases can define lags using a different criteria (e.g. number of messages exceeds a threshold). I think because of these variances, it kinda makes sense for tools like Burrow (and others) to be used for this purpose, instead of having the broker dictating when consumers are lagging. Just my two cents From: dev@kafka.apache.org At: 08/16/22 15:06:16 UTC-4:00To: us...@kafka.apache.org, show...@gmail.com, mmcfarl...@cavulus.com, dev@kafka.apache.org, scante...@gmail.com, ranlupov...@gmail.com, israele...@gmail.com Subject: Re: Consumer Lag-Apache_kafka_JMX metrics Hello Experts, Any info or pointers on my query please. On Mon, Aug 15, 2022 at 11:36 PM Kafka Life wrote: > Dear Kafka Experts > we need to monitor the consumer lag in kafka clusters 2.5.1 and 2.8.0 > versions of kafka in Grafana. > > 1/ What is the correct path for JMX metrics to evaluate Consumer Lag in > kafka cluster. > > 2/ I had thought it is FetcherLag but it looks like it is not as per the > link below. > > https://www.instaclustr.com/support/documentation/kafka/monitoring-information/f etcher-lag-metrics/#:~:text=Aggregated%20Fetcher%20Consumer%20Lag%20This%20metri c%20aggregates%20lag,in%20sync%20with%20partitions%20that%20it%20is%20replicatin g > . > > Could one of you experts please guide on which JMX i should use for > consumer lag apart from kafka burrow or such intermediate tools > > Thanking you in advance > >
Re:[DISCUSS] KIP-848: The Next Generation of the Consumer Rebalance Protocol
Kudos David, Guozhang, and Jason for putting together such a great proposal. I don't want to hijack the discussion, just wanted to mention that it would be great if the final design is made extensible enough, so other use cases (like Kafka Connect, Schema Registry, etc.) can be added later on. I can see how the concept of different group "types" in the group coordinator can be leveraged to support such cases. On KIP-795, I wanted to add public APIs for the AbstractCoordinator with the intent of formalizing the use of the Group Membership Protocol for resource management use cases. I'll probably close this KIP and wait to see what comes out of this redesign of the protocol. Thanks - https://cwiki.apache.org/confluence/display/KAFKA/KIP-795%3A+Add+public+APIs+for+AbstractCoordinator From: dev@kafka.apache.org At: 07/06/22 04:44:59 UTC-4:00To: dev@kafka.apache.org Subject: [DISCUSS] KIP-848: The Next Generation of the Consumer Rebalance Protocol Hi all, I would like to start a discussion thread on KIP-848: The Next Generation of the Consumer Rebalance Protocol. With this KIP, we aim to make the rebalance protocol (for consumers) more reliable, more scalable, easier to implement for clients, and easier to debug for operators. The KIP is here: https://cwiki.apache.org/confluence/x/HhD1D. Please take a look and let me know what you think. Best, David PS: I will be away from July 18th to August 8th. That gives you a bit of time to read and digest this long KIP.
Re: [DISCUSS] KIP-795: Add public APIs for AbstractCoordinator
Thanks David for reading the KIP and sharing your thoughts. I still believe the Group Membership protocol has enormous potential to be used outside of Core Kafka. Even if AbstractCordinator is, as you suggested, not the right abstraction to build upon, there's still merit in the idea of having the group membership protocol mechanisms surfaced into APIs that can be used by applications (like Connect and Streams do today) that interact with Kafka one way or another. I'll think about this a bit longer. Thanks again for your feedback. Regards, From: dja...@confluent.io.INVALID At: 02/01/22 05:01:41 UTC-5:00To: Hector Geraldino (BLOOMBERG/ 919 3RD A ) , dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-795: Add public APIs for AbstractCoordinator Hi Hector, Thanks for the KIP. I finally had a bit of time to read it. I understand that a few services have been leveraging Kafka's Group Membership Protocol to do leader election and/or service discovery. However, I am not entirely convinced that Kafka should be used in that way because specialized systems already exist to solve those use cases. That's just my personal opinion. Assuming that we would want to support such cases in the future, I think that AbstractCoordinator is not the right abstraction for this. AbstractCoordinator is more and internal implementation detail and not really something meant to become a public API. I think that it would be much better to come up with a new set of APIs for these use cases instead of extracting an interface out of AbstractCoordinator. Best, David On Thu, Nov 11, 2021 at 11:15 PM Hector Geraldino (BLOOMBERG/ 919 3RD A) wrote: > > Hi Tom, > > Thanks for taking time reviewing the KIP. > > I think it's reasonable to ask if Kafka's Group Coordination protocol should be used for use cases other than the distributed event log. This was actually briefly addressed by Gwen Shapira during her presentation at the strangeloop conference in '18 (a link to the video is included in the KIP), in which she explain in greater details the protocol internals. > > We should also keep in mind that this protocol is already being used for other use cases outside of core Kafka: Confluent Schema Registry uses it to determine leadership between members of a cluster, Kafka Connect uses it for task assignments, same with Kafka Stream for partition and task distributions, and so on. So having a public, stable API not just for new use cases (like ours) but existing ones is IMHO a good thing to have. I'll amend the KIP and add a bit more details to the motivation and alternatives sections, so the usefulness of this KIP is better understood. > > Now, for the first point of your technical observations (regarding protocolTypes()), I don't think it matters in this context, as the protocol name and subtype are only relevant in the context of a consumer group and group rebalance. It really doesn't matter if two different libraries decide to name their protocols the same. > > For item #2, I was under the impression that, because these classes all implement the org.apache.kafka.common.protocol.[Message, ApiMessage] interface, they are implicitly part of the Kafka protocol and the top-level API. Isn't that really the case? > > And finally, for #3, the goal I had in mind when creating this KPI was a small one: to provide an interface that users can rely on when extending the AbstactCoordinator. So my thought was that, while the AbstractCoordinator itself uses some internal APIs (like ConsumerNetworkClient, ConsumerMetadata and so on) those can remain internal. But it probably makes sense to at least explore the possibility of moving the whole AbstractCoordinator class to be part of the public API. I'll do that exercise, see what it entails, and update the KIP with my findings. > > > Thanks again! > Hector > > > From: dev@kafka.apache.org At: 11/10/21 06:43:59 UTC-5:00To: Hector Geraldino (BLOOMBERG/ 919 3RD A ) , dev@kafka.apache.org > Subject: Re: [DISCUSS] KIP-795: Add public APIs for AbstractCoordinator > > Hi Hector, > > Thanks for the KIP. > > At a high level, I think the question to be answered by the community is > "Should Kafka really be providing this kind of cluster management API?". > While Kafka clients need this to provide their functionality it's a > different thing to expose that as a public API of the project, which is > otherwise about providing a distributed event log/data streaming > platform/. Having a public > API brings a significant commitment for API compatibility, which could > impair the ability of the project to change the API in order to make > improvements to the Kafka clients. The current AbstractCoordinator not > being a supported API means we don't currently have to reason about > compatibility here. So I think it would help the motivation section
Re: [DISCUSS] KIP-814: Static membership protocol should let the leader skip assignment
Hi David, Is the idea here to skip calling performAssignment(...) on the AbstractCoordinator.onJoinLeader(...) method, or adding a new boolean parameter to the performAssignment(...) method? The reason I ask is because I raised KIP-795 a few weeks back, which aims to add a public API for AbstractCoordinator, which might change (or not) with this KIP. I see you also mentioned there's some discussions regarding a new consumer protocol. Is this being discussed somewhere else? I'm curious to know how would it work with other systems (like Kafka Connect or Schema Registry) that rely on the rebalance protocol to handle resource assignments. Apologies in advance if these questions are off-topic for the discussion at hand. Regards, Hector From: dev@kafka.apache.org At: 01/24/22 09:08:58 UTC-5:00To: dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-814: Static membership protocol should let the leader skip assignment Hey Jason, Thanks for your comments. Regarding your first point. Yes, you have it right. Let me complement the KIP to be clearer. Regarding your second point. That is right. New partitions would not be detected while the leader is down. It is definitely not ideal but that seems acceptable to me, at least as a first step. Adding partitions to a topic is an infrequent event so the likelihood of having it while the leader is down is rather low but that could happen. The only way to not suffer from this would be to monitor the metadata changes on the broker side. This implies that we would parse both the subscriptions and the assignments in order to have the full list of topics. I am not sure that it is worth doing it at the moment given that we are thinking about a new consumer protocol. What do you think? I suppose that we would need both in the long term as the current protocol is a bit weird at the moment so we need to fix it anyway. We could use this KIP to fix the protocol and do a subsequent KIP in the future for the server side monitoring if we need it. Best, David On Fri, Jan 21, 2022 at 7:51 PM Jason Gustafson wrote: > > Hey David, > > Thanks for the proposal. This was a tricky bug and I think your approach is > probably the best way forward. > > It would be helpful to add a little more detail to the proposal. When the > coordinator detects that the static leader is returning, it will set > `skipAssignment` to true in the `JoinGroup` response. I believe the intent > is to return all member subscriptions in this response so that the leader > can monitor all topics subscribed in the group (which might be different > from the consumer's own subscription). The leader will then send an empty > `SyncGroup` request to collect its own assignment. Do I have that right? > > I think there might still be an edge case in this proposal (assuming I've > understood it correctly). In between the time that the leader shuts down > and is restarted, it is possible that new partitions are added to one of > the subscribed topics. The returning leader would not know about it > because it has no way to collect the full assignment. Do you think this is > a problem? > > Thanks, > Jason > > On Wed, Jan 19, 2022 at 7:27 AM David Jacot wrote: > > > Hi folks, > > > > I'd like to start a discussion for KIP-814: Static membership protocol > > should let the > > leader skip assignment. This is a small extension to the static > > membership protocol > > to address KAFKA-13435. > > > > The KIP is here: https://cwiki.apache.org/confluence/x/C5-kCw. > > > > Please let me know what you think. > > > > Best, > > David > >
Re: [DISCUSS] KIP-795: Add public APIs for AbstractCoordinator
Hi kafka-devs, I would like a second review to the proposed changes on KIP-795: Add public APIs for AbstractCoordinator [ https://cwiki.apache.org/confluence/display/KAFKA/KIP-795%3A+Add+public+APIs+for +AbstractCoordinator ] I've amended the KIP addressing Tom's feedback, and also opened a PR with the proposed changes [ https://github.com/apache/kafka/pull/11515 ]. There's also a PR for the kafka-monitor tool [ https://github.com/linkedin/kafka-monitor/pull/355 ] that demonstrates how we leveraged AbstractCoordinator to implement a High-Availability mode for it, with the intention of showing how this feature can be leveraged by other services within the Kafka ecosystem and outside it. Thanks for your time and consideration. Hector From: dev@kafka.apache.org At: 11/29/21 13:31:26 UTC-5:00To: dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-795: Add public APIs for AbstractCoordinator Hello again Tom, kafka devs First, congrats on becoming a PMC member! That's so cool. Since your last reply I've updated the KIP trying to address some of your suggestions. A few more details have been added to the motivation section, and also went ahead and opened a draft pull-request with the changes I think are needed for this KIP, with the hope that it makes it easier to discuss the general approach and any other concerns the community may have. KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-795%3A+Add+public+APIs+for +AbstractCoordinator PR: https://github.com/apache/kafka/pull/11515 Looking forward for some community feedback. Regards, Hector From: dev@kafka.apache.org At: 11/11/21 17:15:17 UTC-5:00To: dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-795: Add public APIs for AbstractCoordinator Hi Tom, Thanks for taking time reviewing the KIP. I think it's reasonable to ask if Kafka's Group Coordination protocol should be used for use cases other than the distributed event log. This was actually briefly addressed by Gwen Shapira during her presentation at the strangeloop conference in '18 (a link to the video is included in the KIP), in which she explain in greater details the protocol internals. We should also keep in mind that this protocol is already being used for other use cases outside of core Kafka: Confluent Schema Registry uses it to determine leadership between members of a cluster, Kafka Connect uses it for task assignments, same with Kafka Stream for partition and task distributions, and so on. So having a public, stable API not just for new use cases (like ours) but existing ones is IMHO a good thing to have. I'll amend the KIP and add a bit more details to the motivation and alternatives sections, so the usefulness of this KIP is better understood. Now, for the first point of your technical observations (regarding protocolTypes()), I don't think it matters in this context, as the protocol name and subtype are only relevant in the context of a consumer group and group rebalance. It really doesn't matter if two different libraries decide to name their protocols the same. For item #2, I was under the impression that, because these classes all implement the org.apache.kafka.common.protocol.[Message, ApiMessage] interface, they are implicitly part of the Kafka protocol and the top-level API. Isn't that really the case? And finally, for #3, the goal I had in mind when creating this KPI was a small one: to provide an interface that users can rely on when extending the AbstactCoordinator. So my thought was that, while the AbstractCoordinator itself uses some internal APIs (like ConsumerNetworkClient, ConsumerMetadata and so on) those can remain internal. But it probably makes sense to at least explore the possibility of moving the whole AbstractCoordinator class to be part of the public API. I'll do that exercise, see what it entails, and update the KIP with my findings. Thanks again! Hector From: dev@kafka.apache.org At: 11/10/21 06:43:59 UTC-5:00To: Hector Geraldino (BLOOMBERG/ 919 3RD A ) , dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-795: Add public APIs for AbstractCoordinator Hi Hector, Thanks for the KIP. At a high level, I think the question to be answered by the community is "Should Kafka really be providing this kind of cluster management API?". While Kafka clients need this to provide their functionality it's a different thing to expose that as a public API of the project, which is otherwise about providing a distributed event log/data streaming platform/. Having a public API brings a significant commitment for API compatibility, which could impair the ability of the project to change the API in order to make improvements to the Kafka clients. The current AbstractCoordinator not being a supported API means we don't currently have to reason about compatibility here. So I think it would help the motivation section of the KIP to describe in a bit more detail the use case(s) you have for implementing your own co
Re: [DISCUSS] KIP-795: Add public APIs for AbstractCoordinator
Hello again Tom, kafka devs First, congrats on becoming a PMC member! That's so cool. Since your last reply I've updated the KIP trying to address some of your suggestions. A few more details have been added to the motivation section, and also went ahead and opened a draft pull-request with the changes I think are needed for this KIP, with the hope that it makes it easier to discuss the general approach and any other concerns the community may have. KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-795%3A+Add+public+APIs+for+AbstractCoordinator PR: https://github.com/apache/kafka/pull/11515 Looking forward for some community feedback. Regards, Hector From: dev@kafka.apache.org At: 11/11/21 17:15:17 UTC-5:00To: dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-795: Add public APIs for AbstractCoordinator Hi Tom, Thanks for taking time reviewing the KIP. I think it's reasonable to ask if Kafka's Group Coordination protocol should be used for use cases other than the distributed event log. This was actually briefly addressed by Gwen Shapira during her presentation at the strangeloop conference in '18 (a link to the video is included in the KIP), in which she explain in greater details the protocol internals. We should also keep in mind that this protocol is already being used for other use cases outside of core Kafka: Confluent Schema Registry uses it to determine leadership between members of a cluster, Kafka Connect uses it for task assignments, same with Kafka Stream for partition and task distributions, and so on. So having a public, stable API not just for new use cases (like ours) but existing ones is IMHO a good thing to have. I'll amend the KIP and add a bit more details to the motivation and alternatives sections, so the usefulness of this KIP is better understood. Now, for the first point of your technical observations (regarding protocolTypes()), I don't think it matters in this context, as the protocol name and subtype are only relevant in the context of a consumer group and group rebalance. It really doesn't matter if two different libraries decide to name their protocols the same. For item #2, I was under the impression that, because these classes all implement the org.apache.kafka.common.protocol.[Message, ApiMessage] interface, they are implicitly part of the Kafka protocol and the top-level API. Isn't that really the case? And finally, for #3, the goal I had in mind when creating this KPI was a small one: to provide an interface that users can rely on when extending the AbstactCoordinator. So my thought was that, while the AbstractCoordinator itself uses some internal APIs (like ConsumerNetworkClient, ConsumerMetadata and so on) those can remain internal. But it probably makes sense to at least explore the possibility of moving the whole AbstractCoordinator class to be part of the public API. I'll do that exercise, see what it entails, and update the KIP with my findings. Thanks again! Hector From: dev@kafka.apache.org At: 11/10/21 06:43:59 UTC-5:00To: Hector Geraldino (BLOOMBERG/ 919 3RD A ) , dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-795: Add public APIs for AbstractCoordinator Hi Hector, Thanks for the KIP. At a high level, I think the question to be answered by the community is "Should Kafka really be providing this kind of cluster management API?". While Kafka clients need this to provide their functionality it's a different thing to expose that as a public API of the project, which is otherwise about providing a distributed event log/data streaming platform/. Having a public API brings a significant commitment for API compatibility, which could impair the ability of the project to change the API in order to make improvements to the Kafka clients. The current AbstractCoordinator not being a supported API means we don't currently have to reason about compatibility here. So I think it would help the motivation section of the KIP to describe in a bit more detail the use case(s) you have for implementing your own coordinators. For example, are these applications using Kafka otherwise, or just to leverage this API? And what alternatives to implementing your own coordinators did you consider, and why did you reject them? From a technical point of view, there are a number of issues I think would need addressing in order to do something like this: 1. There probably ought to be a way to ensure that protocolTypes() don't collide, or at least reduce the chances of a collision. While probably unlikely in practice the consequences of different protocols having the same name could be pretty confusing to debug. 2. JoinGroupRequestData and JoinGroupResponseData are not public classes (none of the *RequestData or *ResponseData classes are, intentionally), so there would have to be an abstraction for them. 3. It's all well and good having an interface that anyone can implement, but there is no supported Kafka API w
Re: [DISCUSS] KIP-795: Add public APIs for AbstractCoordinator
Hi Tom, Thanks for taking time reviewing the KIP. I think it's reasonable to ask if Kafka's Group Coordination protocol should be used for use cases other than the distributed event log. This was actually briefly addressed by Gwen Shapira during her presentation at the strangeloop conference in '18 (a link to the video is included in the KIP), in which she explain in greater details the protocol internals. We should also keep in mind that this protocol is already being used for other use cases outside of core Kafka: Confluent Schema Registry uses it to determine leadership between members of a cluster, Kafka Connect uses it for task assignments, same with Kafka Stream for partition and task distributions, and so on. So having a public, stable API not just for new use cases (like ours) but existing ones is IMHO a good thing to have. I'll amend the KIP and add a bit more details to the motivation and alternatives sections, so the usefulness of this KIP is better understood. Now, for the first point of your technical observations (regarding protocolTypes()), I don't think it matters in this context, as the protocol name and subtype are only relevant in the context of a consumer group and group rebalance. It really doesn't matter if two different libraries decide to name their protocols the same. For item #2, I was under the impression that, because these classes all implement the org.apache.kafka.common.protocol.[Message, ApiMessage] interface, they are implicitly part of the Kafka protocol and the top-level API. Isn't that really the case? And finally, for #3, the goal I had in mind when creating this KPI was a small one: to provide an interface that users can rely on when extending the AbstactCoordinator. So my thought was that, while the AbstractCoordinator itself uses some internal APIs (like ConsumerNetworkClient, ConsumerMetadata and so on) those can remain internal. But it probably makes sense to at least explore the possibility of moving the whole AbstractCoordinator class to be part of the public API. I'll do that exercise, see what it entails, and update the KIP with my findings. Thanks again! Hector From: dev@kafka.apache.org At: 11/10/21 06:43:59 UTC-5:00To: Hector Geraldino (BLOOMBERG/ 919 3RD A ) , dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-795: Add public APIs for AbstractCoordinator Hi Hector, Thanks for the KIP. At a high level, I think the question to be answered by the community is "Should Kafka really be providing this kind of cluster management API?". While Kafka clients need this to provide their functionality it's a different thing to expose that as a public API of the project, which is otherwise about providing a distributed event log/data streaming platform/. Having a public API brings a significant commitment for API compatibility, which could impair the ability of the project to change the API in order to make improvements to the Kafka clients. The current AbstractCoordinator not being a supported API means we don't currently have to reason about compatibility here. So I think it would help the motivation section of the KIP to describe in a bit more detail the use case(s) you have for implementing your own coordinators. For example, are these applications using Kafka otherwise, or just to leverage this API? And what alternatives to implementing your own coordinators did you consider, and why did you reject them? From a technical point of view, there are a number of issues I think would need addressing in order to do something like this: 1. There probably ought to be a way to ensure that protocolTypes() don't collide, or at least reduce the chances of a collision. While probably unlikely in practice the consequences of different protocols having the same name could be pretty confusing to debug. 2. JoinGroupRequestData and JoinGroupResponseData are not public classes (none of the *RequestData or *ResponseData classes are, intentionally), so there would have to be an abstraction for them. 3. It's all well and good having an interface that anyone can implement, but there is no supported Kafka API which takes an instance as a parameter (i.e. where do you plug your implementation in without having to use a bunch of other non-public Kafka APIs?) I assume in your current usage you're having to make use of other non-supported client APIs to make use of your coordinator. The KIP isn't really a complete piece of work without a way to use a custom implementation, in my opinion. It would be confusing if it looked like we were encouraging people to use those other non-supported APIs by making this coordinator public. Kind regards, Tom On Mon, Nov 8, 2021 at 2:01 PM Hector Geraldino (BLOOMBERG/ 919 3RD A) < hgerald...@bloomberg.net> wrote: > Hi Kafka devs, > > I would like to start the discussion of KIP-795: Add public APIs for > AbstractCoordinator > > > https://cwiki.apache.org/confluence/dis
[DISCUSS] KIP-795: Add public APIs for AbstractCoordinator
Hi Kafka devs, I would like to start the discussion of KIP-795: Add public APIs for AbstractCoordinator https://cwiki.apache.org/confluence/display/KAFKA/KIP-795%3A+Add+public+APIs+for+AbstractCoordinator Looking forward for some feedback from the community. Regards, Hector
DISMISS - Re:[DISCUSS] KIP-784: Add public APIs for AbstractCoordinator
Please dismiss this message, as the KIP number is wrong. I'll send a new message with the correct KIP shortly. Apologies From: dev@kafka.apache.org At: 11/08/21 08:45:22 UTC-5:00To: dev@kafka.apache.org Subject: [DISCUSS] KIP-784: Add public APIs for AbstractCoordinator Hi Kafka devs, I would like to start the discussion of KIP-784: Add public APIs for AbstractCoordinator https://cwiki.apache.org/confluence/display/KAFKA/KIP-784%3A+Add+public+APIs+for +AbstractCoordinator Looking forward for some feedback from the community. Regards, Hector
[DISCUSS] KIP-784: Add public APIs for AbstractCoordinator
Hi Kafka devs, I would like to start the discussion of KIP-784: Add public APIs for AbstractCoordinator https://cwiki.apache.org/confluence/display/KAFKA/KIP-784%3A+Add+public+APIs+for+AbstractCoordinator Looking forward for some feedback from the community. Regards, Hector
Wiki Permissions
Hello, I'd like to be added to the contributors list, so I can submit a KIP. My Jira ID is: hgeraldino Wiki ID: hgeraldino Thanks, Hector
