Re: KIP-919: Allow AdminClient to Talk Directly with the KRaft Controller Quorum

2024-04-11 Thread Nelson B.
Hi Colin, Thanks for the KIP! I'm sorry for reopening this old discussion thread but I don't know where else I can ask my question. I wanted to use the newly updated kafka-configs.sh tool to dynamically update the ssl keystore in the controller node, but I still couldn't figure it out. Is it

Re: [VOTE] KIP-919: Allow AdminClient to Talk Directly with the KRaft Controller Quorum and add Controller Registration

2023-07-27 Thread Ismael Juma
ote: > >> > >> > >> > >> Thanks for driving this KIP, Colin! > >> > >> > >> > >> +1 binding > >> > >> > >> > >> -David > >> > >> > >> > >>

Re: [VOTE] KIP-919: Allow AdminClient to Talk Directly with the KRaft Controller Quorum and add Controller Registration

2023-07-27 Thread Colin McCabe
t; >> > >> On Wed, Jul 26, 2023 at 8:58 AM Divij Vaidya >> > >> wrote: >> > >> >> > >> > +1 (binding) >> > >> > >> > >> > -- >> > >> > Divij Vaidya >> > >> > >>

Re: [VOTE] KIP-919: Allow AdminClient to Talk Directly with the KRaft Controller Quorum and add Controller Registration

2023-07-27 Thread Ron Dagostino
> > >> On Wed, Jul 26, 2023 at 8:58 AM Divij Vaidya > > >> wrote: > > >> > > >> > +1 (binding) > > >> > > > >> > -- > > >> > Divij Vaidya > > >> > > > >> > > > >> > On Wed, Jul 26

Re: [VOTE] KIP-919: Allow AdminClient to Talk Directly with the KRaft Controller Quorum and add Controller Registration

2023-07-27 Thread Ismael Juma
e: > >> > > > >> > > +1 (binding) from me. > >> > > > >> > > Thanks for the KIP! > >> > > > >> > > -- > >> > > Ziming > >> > > > >> > > > On Jul 26, 2023,

Re: [VOTE] KIP-919: Allow AdminClient to Talk Directly with the KRaft Controller Quorum and add Controller Registration

2023-07-26 Thread Colin McCabe
> wrote: >>> > > >>> > > +1 (binding) from me. >>> > > >>> > > Thanks for the KIP! >>> > > >>> > > -- >>> > > Ziming >>> > > >>> > > > On Jul 26, 202

Re: [VOTE] KIP-919: Allow AdminClient to Talk Directly with the KRaft Controller Quorum and add Controller Registration

2023-07-26 Thread Colin McCabe
> > >> > > -- >> > > Ziming >> > > >> > > > On Jul 26, 2023, at 20:18, Luke Chen wrote: >> > > > >> > > > +1 (binding) from me. >> > > > >> > > > Thanks for the KIP! >> > > > >

Re: [VOTE] KIP-919: Allow AdminClient to Talk Directly with the KRaft Controller Quorum and add Controller Registration

2023-07-26 Thread Ron Dagostino
t; > > Ziming > > > > > > > On Jul 26, 2023, at 20:18, Luke Chen wrote: > > > > > > > > +1 (binding) from me. > > > > > > > > Thanks for the KIP! > > > > > > > > Luke > > > > > > > > On Tue, Jul

Re: [VOTE] KIP-919: Allow AdminClient to Talk Directly with the KRaft Controller Quorum and add Controller Registration

2023-07-26 Thread David Arthur
; > Thanks for the KIP! > > > > -- > > Ziming > > > > > On Jul 26, 2023, at 20:18, Luke Chen wrote: > > > > > > +1 (binding) from me. > > > > > > Thanks for the KIP! > > > > > > Luke > > > > > > On

Re: [VOTE] KIP-919: Allow AdminClient to Talk Directly with the KRaft Controller Quorum and add Controller Registration

2023-07-26 Thread Divij Vaidya
hanks for the KIP! > > > > Luke > > > > On Tue, Jul 25, 2023 at 1:24 AM Colin McCabe wrote: > > > >> Hi all, > >> > >> I'd like to start the vote for KIP-919: Allow AdminClient to Talk Directly > >> with the KRaft Controller Quorum a

Re: [VOTE] KIP-919: Allow AdminClient to Talk Directly with the KRaft Controller Quorum and add Controller Registration

2023-07-26 Thread ziming deng
t; I'd like to start the vote for KIP-919: Allow AdminClient to Talk Directly >> with the KRaft Controller Quorum and add Controller Registration. >> >> The KIP is here: https://cwiki.apache.org/confluence/x/Owo0Dw >> >> Thanks to everyone who reviewed the proposal. >> >> best, >> Colin >>

Re: [VOTE] KIP-919: Allow AdminClient to Talk Directly with the KRaft Controller Quorum and add Controller Registration

2023-07-26 Thread Luke Chen
+1 (binding) from me. Thanks for the KIP! Luke On Tue, Jul 25, 2023 at 1:24 AM Colin McCabe wrote: > Hi all, > > I'd like to start the vote for KIP-919: Allow AdminClient to Talk Directly > with the KRaft Controller Quorum and add Controller Registration. > > The

Re: KIP-919: Allow AdminClient to Talk Directly with the KRaft Controller Quorum

2023-07-25 Thread Colin McCabe
On Tue, Jul 25, 2023, at 05:30, Luke Chen wrote: > Hi Colin, > > Some more comments: > 1. In the KIP, we mentioned "controller heartbeats", but it is not > explained anywhere. > I think "controller heartbeats" = controller registration", is that > correct? > If no, please explain more about it in

Re: KIP-919: Allow AdminClient to Talk Directly with the KRaft Controller Quorum

2023-07-25 Thread Luke Chen
Hi Colin, Some more comments: 1. In the KIP, we mentioned "controller heartbeats", but it is not explained anywhere. I think "controller heartbeats" = controller registration", is that correct? If no, please explain more about it in the KIP. 2. Following this question: > Which endpoint will the

[VOTE] KIP-919: Allow AdminClient to Talk Directly with the KRaft Controller Quorum and add Controller Registration

2023-07-24 Thread Colin McCabe
Hi all, I'd like to start the vote for KIP-919: Allow AdminClient to Talk Directly with the KRaft Controller Quorum and add Controller Registration. The KIP is here: https://cwiki.apache.org/confluence/x/Owo0Dw Thanks to everyone who reviewed the proposal. best, Colin

Re: KIP-919: Allow AdminClient to Talk Directly with the KRaft Controller Quorum

2023-07-21 Thread Colin McCabe
On Fri, Jul 21, 2023, at 09:43, José Armando García Sancio wrote: > Thanks for the KIP Colin. Apologies if some of these points have > already been made. I have not followed the discussion closely: > > 1. Re: Periodically, each controller will check that the controller > registration for its ID is

Re: KIP-919: Allow AdminClient to Talk Directly with the KRaft Controller Quorum

2023-07-21 Thread Colin McCabe
On Fri, Jul 21, 2023, at 08:14, David Arthur wrote: > Hey Colin, thanks for the KIP! Some questions > Hi David, Thanks for reviewing. > 1) "This registration will include information about the endpoints which > they possess" Will this include all endpoints, or only those configured in >

Re: KIP-919: Allow AdminClient to Talk Directly with the KRaft Controller Quorum

2023-07-21 Thread José Armando García Sancio
Thanks for the KIP Colin. Apologies if some of these points have already been made. I have not followed the discussion closely: 1. Re: Periodically, each controller will check that the controller registration for its ID is as expected Does this need to be periodic? Can't the controller schedule

Re: KIP-919: Allow AdminClient to Talk Directly with the KRaft Controller Quorum

2023-07-21 Thread David Arthur
Hey Colin, thanks for the KIP! Some questions 1) "This registration will include information about the endpoints which they possess" Will this include all endpoints, or only those configured in "advertised.listeners" 2) "Periodically, each controller will check that the controller registration

Re: KIP-919: Allow AdminClient to Talk Directly with the KRaft Controller Quorum

2023-07-20 Thread Colin McCabe
On Tue, Jul 18, 2023, at 09:30, Mickael Maison wrote: > H Colin, > > Thanks for the KIP. > > Just a few points: > 1. As Tom mentioned it would be good to clarify the APIs we expect > available on controllers. I assume we want to add DESCRIBE_CONFIGS as > part of this KIP. Hi Mickael, Yes, this

Re: KIP-919: Allow AdminClient to Talk Directly with the KRaft Controller Quorum

2023-07-20 Thread Colin McCabe
On Tue, Jul 4, 2023, at 05:19, Luke Chen wrote: > 4. Does all the command-line tools with `--bootstrap-controllers` support > all the options in the tool? > For example, kafka-configs.sh, In addition to the `--alter` option you > mentioned in the example, do we also support `--describe` or

Re: KIP-919: Allow AdminClient to Talk Directly with the KRaft Controller Quorum

2023-07-20 Thread Colin McCabe
On Tue, Jul 4, 2023, at 04:46, Tom Bentley wrote: > Hi Colin, > > Thanks for the KIP. > > 1. It mentions kafka-configs.sh as one of the affected tools, but doesn't > mention that ControllerApis doesn't currently support DESCRIBE_CONFIGS. I > think this is worth noting as it is, in effect, a change

Re: KIP-919: Allow AdminClient to Talk Directly with the KRaft Controller Quorum

2023-07-18 Thread Mickael Maison
H Colin, Thanks for the KIP. Just a few points: 1. As Tom mentioned it would be good to clarify the APIs we expect available on controllers. I assume we want to add DESCRIBE_CONFIGS as part of this KIP. 2. Currently we have no way of retrieving the list of configs that apply to controllers. It

Re: KIP-919: Allow AdminClient to Talk Directly with the KRaft Controller Quorum

2023-07-04 Thread Luke Chen
Hi Colin, Thanks for the answers to my previous questions. > Yes, the common thread here is that all of these shell commands perform operations can be done without the broker. So it's reasonable to allow them to be done without going through the broker. I don't know if we need a separate note

Re: KIP-919: Allow AdminClient to Talk Directly with the KRaft Controller Quorum

2023-07-04 Thread Tom Bentley
Hi Colin, Thanks for the KIP. 1. It mentions kafka-configs.sh as one of the affected tools, but doesn't mention that ControllerApis doesn't currently support DESCRIBE_CONFIGS. I think this is worth noting as it is, in effect, a change to the wire protocol accepted by the controller, even if it's

Re: KIP-919: Allow AdminClient to Talk Directly with the KRaft Controller Quorum

2023-04-27 Thread Colin McCabe
On Wed, Apr 26, 2023, at 22:08, Luke Chen wrote: > Hi Colin, > > Some comments: > 1. I agree we should set "top-level" errors for metadata response > > 2. In the "brokers" field of metadata response from controller, it'll > respond with "Controller endpoint information as given in >

Re: KIP-919: Allow AdminClient to Talk Directly with the KRaft Controller Quorum

2023-04-26 Thread Luke Chen
Hi Colin, Some comments: 1. I agree we should set "top-level" errors for metadata response 2. In the "brokers" field of metadata response from controller, it'll respond with "Controller endpoint information as given in controller.quorum.voters", instead of the "alive" controllers(voters). That

Re: KIP-919: Allow AdminClient to Talk Directly with the KRaft Controller Quorum

2023-04-26 Thread Colin McCabe
On Tue, Apr 25, 2023, at 04:59, Divij Vaidya wrote: > Thank you for the KIP Colin. > > In general, I like the idea of having the ability to interact directly with > the controllers. I agree with your observation that it helps in situations > where you would want to get data directly from the

Re: KIP-919: Allow AdminClient to Talk Directly with the KRaft Controller Quorum

2023-04-25 Thread Divij Vaidya
Thank you for the KIP Colin. In general, I like the idea of having the ability to interact directly with the controllers. I agree with your observation that it helps in situations where you would want to get data directly from the controller instead of going via a broker. I have some general

Re: KIP-919: Allow AdminClient to Talk Directly with the KRaft Controller Quorum

2023-04-24 Thread Colin McCabe
On Fri, Apr 21, 2023, at 14:17, Jason Gustafson wrote: > Hey Colin, > > The KIP makes sense overall. Nice to clarify the contract between clients > and the controllers. The use of `DirectToKRaftControllerQuorum` will help > prevent misconfiguration. In fact, I wonder if we can return a fatal error

Re: KIP-919: Allow AdminClient to Talk Directly with the KRaft Controller Quorum

2023-04-21 Thread Jason Gustafson
Hey Colin, The KIP makes sense overall. Nice to clarify the contract between clients and the controllers. The use of `DirectToKRaftControllerQuorum` will help prevent misconfiguration. In fact, I wonder if we can return a fatal error instead of NOT_CONTROLLER so that the client would immediately

Re: KIP-919: Allow AdminClient to Talk Directly with the KRaft Controller Quorum

2023-04-20 Thread Colin McCabe
On Wed, Apr 19, 2023, at 20:56, Philip Nee wrote: > Hey Colin, > > I still need to finish reading and understanding the KIP, but I have a > couple of comments despite being ignorant of most of the KRaft stuff. > (Sorry!) > > Firstly, does it make sense to create an extension of the current >

Re: KIP-919: Allow AdminClient to Talk Directly with the KRaft Controller Quorum

2023-04-19 Thread Philip Nee
Hey Colin, I still need to finish reading and understanding the KIP, but I have a couple of comments despite being ignorant of most of the KRaft stuff. (Sorry!) Firstly, does it make sense to create an extension of the current AdminClient only to handle these specific KRaft use cases? It seems

Re: KIP-919: Allow AdminClient to Talk Directly with the KRaft Controller Quorum

2023-04-19 Thread ziming deng
Hello Colin, There is a mistake that we use `—bootstrap-server` instead of `—bootstrap-server(s)`, so should we also change the new argument `—bootstrap-controller` (no s). -- Ziming > On Apr 20, 2023, at 05:17, Colin McCabe wrote: > > Hi all, > > I wrote a short KIP about allowing

Re: KIP-919: Allow AdminClient to Talk Directly with the KRaft Controller Quorum

2023-04-19 Thread Colin McCabe
On Wed, Apr 19, 2023, at 14:37, Ron Dagostino wrote: > Thanks for the KIP, Colin. > > There seems to be some inconsistency between sometimes referring to > "TargetKRaftControllerQuorum" and other times referring to > "DirectToKRaftControllerQuorum". Aside from that, it looks good to > me. The

Re: KIP-919: Allow AdminClient to Talk Directly with the KRaft Controller Quorum

2023-04-19 Thread Ron Dagostino
Thanks for the KIP, Colin. There seems to be some inconsistency between sometimes referring to "TargetKRaftControllerQuorum" and other times referring to "DirectToKRaftControllerQuorum". Aside from that, it looks good to me. The symmetry of bootstrap servers and bootstrap controllers feels

KIP-919: Allow AdminClient to Talk Directly with the KRaft Controller Quorum

2023-04-19 Thread Colin McCabe
Hi all, I wrote a short KIP about allowing AdminClient to talk directly with the KRaft controller quorum. Check it out here: https://cwiki.apache.org/confluence/x/Owo0Dw best, Colin