Re: [DISCUSS] KIP-896: Remove old client protocol API versions in Kafka 4.0
Sorry, the underscore was meant to refer to the https://github.com/apache/kafka/tree/trunk/clients/src/main/resources/common/message directory in the previous message. Den fre 24 nov. 2023 kl 14:30 skrev Anton Agestam : > Hi Ismael, > > This looks like a healthy KIP for Kafka > > As the implementer of a Kafka Protocol library for Python, Aiven-Open/kio > [1], I'm curious how this change will affect the library. > > We generate entities for the full protocol by introspecting the JSON > schema definitions under _. How will the KIP change those definitions? Will > the dropped versions be reflected as bumps of the lower limit > in "validVersions"? > > Thanks and BR, > Anton > > [1]: https://github.com/Aiven-Open/kio > > On 2023/01/03 16:17:24 Ismael Juma wrote: > > Hi all, > > > > I would like to start a discussion regarding the removal of very old > client > > protocol API versions in Apache Kafka 4.0 to improve maintainability & > > supportability of Kafka. Please take a look at the proposal: > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-896%3A+Remove+old+client+protocol+API+versions+in+Kafka+4.0 > > > > Ismael > > >
Re: [DISCUSS] KIP-896: Remove old client protocol API versions in Kafka 4.0
Hi Anton, The json schema definition files haven't been deemed public api yet although they perhaps should (KIPs welcome). Your suggestion to update `validVersions` sounds good to me. Longer-term, it would also make sense to consider a `deprecatedVersions` field (this is especially useful if we deprecate at the start of a new major versions cycle, but this requires a new KIP). Ismael On Fri, Nov 24, 2023 at 5:31 AM Anton Agestam wrote: > Hi Ismael, > > This looks like a healthy KIP for Kafka > > As the implementer of a Kafka Protocol library for Python, Aiven-Open/kio > [1], I'm curious how this change will affect the library. > > We generate entities for the full protocol by introspecting the JSON schema > definitions under _. How will the KIP change those definitions? Will the > dropped versions be reflected as bumps of the lower limit > in "validVersions"? > > Thanks and BR, > Anton > > [1]: https://github.com/Aiven-Open/kio > > On 2023/01/03 16:17:24 Ismael Juma wrote: > > Hi all, > > > > I would like to start a discussion regarding the removal of very old > client > > protocol API versions in Apache Kafka 4.0 to improve maintainability & > > supportability of Kafka. Please take a look at the proposal: > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-896%3A+Remove+old+client+protocol+API+versions+in+Kafka+4.0 > > > > Ismael > > >
RE: [DISCUSS] KIP-896: Remove old client protocol API versions in Kafka 4.0
Hi Ismael, This looks like a healthy KIP for Kafka As the implementer of a Kafka Protocol library for Python, Aiven-Open/kio [1], I'm curious how this change will affect the library. We generate entities for the full protocol by introspecting the JSON schema definitions under _. How will the KIP change those definitions? Will the dropped versions be reflected as bumps of the lower limit in "validVersions"? Thanks and BR, Anton [1]: https://github.com/Aiven-Open/kio On 2023/01/03 16:17:24 Ismael Juma wrote: > Hi all, > > I would like to start a discussion regarding the removal of very old client > protocol API versions in Apache Kafka 4.0 to improve maintainability & > supportability of Kafka. Please take a look at the proposal: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-896%3A+Remove+old+client+protocol+API+versions+in+Kafka+4.0 > > Ismael >
Re: [DISCUSS] KIP-896: Remove old client protocol API versions in Kafka 4.0
I started a vote thread for this since I addressed the comments so far and it doesn't seem like there were any major concerns. Ismael On Tue, Jan 3, 2023 at 8:17 AM Ismael Juma wrote: > Hi all, > > I would like to start a discussion regarding the removal of very old > client protocol API versions in Apache Kafka 4.0 to improve maintainability > & supportability of Kafka. Please take a look at the proposal: > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-896%3A+Remove+old+client+protocol+API+versions+in+Kafka+4.0 > > Ismael >
Re: [DISCUSS] KIP-896: Remove old client protocol API versions in Kafka 4.0
Hi Jose, I updated the KIP to include a new metric for deprecated request api versions and also a new attribute in the request log to make it easy to find such entries, Thanks, Ismael On Thu, Jan 12, 2023 at 1:03 AM Ismael Juma wrote: > Hi Jose, > > I think it's reasonable to add more user-friendly metrics as you > described. I'll update the KIP soon with that. I'll try to define them in a > way where they track deprecated protocols for the next major release. That > way, they can be useful even after AK 4.0 is released. > > Ismael > > On Wed, Jan 11, 2023 at 12:34 PM José Armando García Sancio > wrote: > >> Thanks Ismael. >> >> > The following metrics are used to determine both questions: >> > > >> > >- Client name and version: >> > > >> kafka.server:clientSoftwareName=(client-software-name),clientSoftwareVersion=(client-software-version),listener=(listener),networkProcessor=(processor-index),type=(type) >> > >- Request name and version: >> > > >> kafka.network:type=RequestMetrics,name=RequestsPerSec,request=(api-name),version=(api-version)} >> > > >> > > >> > Are you suggesting that this is too complicated and hence we should add >> a >> > metric that tracks AK 4.0 support explicitly? >> >> Correct. It doesn't look trivial for the users to implement this check >> against the RequestMetrics. I was wondering if it is worth it for >> Kafka to implement this for them and expose a simple metric that they >> can check. >> >> -- >> -José >> >
Re: [DISCUSS] KIP-896: Remove old client protocol API versions in Kafka 4.0
Hi Jun, I updated the KIP to include the inter-broker protocol version deprecation. Ismael On Mon, Jan 9, 2023 at 3:53 PM Jun Rao wrote: > Hi, Ismael, > > Thanks for the reply. > > 2. Yes, that makes sense. > > Jun > > On Mon, Jan 9, 2023 at 11:26 AM Ismael Juma wrote: > > > Hi Jun, > > > > Thanks for the feedback. > > > > 1. Good question. I was looking at the IBP issue as more of an inter > broker > > protocol communication thing and hence I thought we'd cover that as part > of > > the KIP that modernizes that aspect (yet to be written). I filed a JIRA > > recently that shares some of the thinking: > > https://issues.apache.org/jira/browse/KAFKA-14552. I was hoping we would > > tackle the details of that as part of the zk deprecation/removal. You're > > right though that we need to deprecate IBP before 2.1 as part of this KIP > > since we are removing some of the protocol API versions that would be > > required for that to work. I'll update the KIP to include this. > > > > 2. I believe they are all introduced after 2.1, yes. This is what I used > as > > the source for 2.1: > > > > > https://github.com/apache/kafka/blob/2.1/clients/src/main/java/org/apache/kafka/common/protocol/ApiKeys.java#L189 > > . > > Note that I excluded inter broker protocol APIs like LeaderAndIsr, > > StopReplica and others for the reason I explained in `1`. Does that make > > sense? > > > > Ismael > > > > On Mon, Jan 9, 2023 at 11:01 AM Jun Rao > wrote: > > > > > Hi, Ismael, > > > > > > Thanks for the KIP. A couple of comments. > > > > > > 1. Should we deprecate IBPs before 2.1 too? > > > > > > 2. There seem to be more requests than those listed in the KIP. Are > they > > > all introduced after 2.1? > > > > > > Thanks, > > > > > > Jun > > > > > > > > > > > > On Sat, Jan 7, 2023 at 5:13 PM Ismael Juma wrote: > > > > > > > Hi all, > > > > > > > > I updated the KIP to make it explicit that the Java clients shipped > > with > > > > Apache Kafka 4.0 would only be guaranteed to work with brokers > running > > > > Apache Kafka 2.1 or newer, since they too would no longer have access > > to > > > > the removed protocol APIs. Thanks to Jason for asking about this > aspect > > > > offline. > > > > > > > > I will leave this KIP open for discussion for another couple or weeks > > or > > > > so. Please do share your thoughts given the high impact of the > change. > > > > > > > > Ismael > > > > > > > > On Tue, Jan 3, 2023 at 8:17 AM Ismael Juma > wrote: > > > > > > > > > Hi all, > > > > > > > > > > I would like to start a discussion regarding the removal of very > old > > > > > client protocol API versions in Apache Kafka 4.0 to improve > > > > maintainability > > > > > & supportability of Kafka. Please take a look at the proposal: > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-896%3A+Remove+old+client+protocol+API+versions+in+Kafka+4.0 > > > > > > > > > > Ismael > > > > > > > > > > > > > > >
Re: [DISCUSS] KIP-896: Remove old client protocol API versions in Kafka 4.0
Hi Jose, I think it's reasonable to add more user-friendly metrics as you described. I'll update the KIP soon with that. I'll try to define them in a way where they track deprecated protocols for the next major release. That way, they can be useful even after AK 4.0 is released. Ismael On Wed, Jan 11, 2023 at 12:34 PM José Armando García Sancio wrote: > Thanks Ismael. > > > The following metrics are used to determine both questions: > > > > > >- Client name and version: > > > > kafka.server:clientSoftwareName=(client-software-name),clientSoftwareVersion=(client-software-version),listener=(listener),networkProcessor=(processor-index),type=(type) > > >- Request name and version: > > > > kafka.network:type=RequestMetrics,name=RequestsPerSec,request=(api-name),version=(api-version)} > > > > > > > > Are you suggesting that this is too complicated and hence we should add a > > metric that tracks AK 4.0 support explicitly? > > Correct. It doesn't look trivial for the users to implement this check > against the RequestMetrics. I was wondering if it is worth it for > Kafka to implement this for them and expose a simple metric that they > can check. > > -- > -José >
Re: [DISCUSS] KIP-896: Remove old client protocol API versions in Kafka 4.0
Thanks Ismael. > The following metrics are used to determine both questions: > > > >- Client name and version: > > > > kafka.server:clientSoftwareName=(client-software-name),clientSoftwareVersion=(client-software-version),listener=(listener),networkProcessor=(processor-index),type=(type) > >- Request name and version: > > > > kafka.network:type=RequestMetrics,name=RequestsPerSec,request=(api-name),version=(api-version)} > > > > > Are you suggesting that this is too complicated and hence we should add a > metric that tracks AK 4.0 support explicitly? Correct. It doesn't look trivial for the users to implement this check against the RequestMetrics. I was wondering if it is worth it for Kafka to implement this for them and expose a simple metric that they can check. -- -José
Re: [DISCUSS] KIP-896: Remove old client protocol API versions in Kafka 4.0
Hi Jose, The KIP describes a couple of existing metrics that can be used to answer that question: The following metrics are used to determine both questions: > >- Client name and version: > > kafka.server:clientSoftwareName=(client-software-name),clientSoftwareVersion=(client-software-version),listener=(listener),networkProcessor=(processor-index),type=(type) >- Request name and version: > > kafka.network:type=RequestMetrics,name=RequestsPerSec,request=(api-name),version=(api-version)} > > Are you suggesting that this is too complicated and hence we should add a metric that tracks AK 4.0 support explicitly? Ismael On Tue, Jan 10, 2023 at 10:33 AM José Armando García Sancio wrote: > Hi Ismael, > > Thanks for the improvement. > > I haven't been following the discussion in detail so it is possible > that this was already discussed. > > If a user upgrades to Apache Kafka 4.0 it is possible for some of > their clients to stop working because the request's version would not > be a version that Kafka 4.0 supports. Should we add metrics or some > other mechanism that the user can monitor to determine if it is safe > to upgrade Kafka to 4.0. For example, the metrics could report if a > Kafka broker received a request or response in the past 7 days that > would not be supported by Kafka 4.0. > > Thanks > -- > -José >
Re: [DISCUSS] KIP-896: Remove old client protocol API versions in Kafka 4.0
Hi Ismael, Thanks for the improvement. I haven't been following the discussion in detail so it is possible that this was already discussed. If a user upgrades to Apache Kafka 4.0 it is possible for some of their clients to stop working because the request's version would not be a version that Kafka 4.0 supports. Should we add metrics or some other mechanism that the user can monitor to determine if it is safe to upgrade Kafka to 4.0. For example, the metrics could report if a Kafka broker received a request or response in the past 7 days that would not be supported by Kafka 4.0. Thanks -- -José
Re: [DISCUSS] KIP-896: Remove old client protocol API versions in Kafka 4.0
Hi, Ismael, Thanks for the reply. 2. Yes, that makes sense. Jun On Mon, Jan 9, 2023 at 11:26 AM Ismael Juma wrote: > Hi Jun, > > Thanks for the feedback. > > 1. Good question. I was looking at the IBP issue as more of an inter broker > protocol communication thing and hence I thought we'd cover that as part of > the KIP that modernizes that aspect (yet to be written). I filed a JIRA > recently that shares some of the thinking: > https://issues.apache.org/jira/browse/KAFKA-14552. I was hoping we would > tackle the details of that as part of the zk deprecation/removal. You're > right though that we need to deprecate IBP before 2.1 as part of this KIP > since we are removing some of the protocol API versions that would be > required for that to work. I'll update the KIP to include this. > > 2. I believe they are all introduced after 2.1, yes. This is what I used as > the source for 2.1: > > https://github.com/apache/kafka/blob/2.1/clients/src/main/java/org/apache/kafka/common/protocol/ApiKeys.java#L189 > . > Note that I excluded inter broker protocol APIs like LeaderAndIsr, > StopReplica and others for the reason I explained in `1`. Does that make > sense? > > Ismael > > On Mon, Jan 9, 2023 at 11:01 AM Jun Rao wrote: > > > Hi, Ismael, > > > > Thanks for the KIP. A couple of comments. > > > > 1. Should we deprecate IBPs before 2.1 too? > > > > 2. There seem to be more requests than those listed in the KIP. Are they > > all introduced after 2.1? > > > > Thanks, > > > > Jun > > > > > > > > On Sat, Jan 7, 2023 at 5:13 PM Ismael Juma wrote: > > > > > Hi all, > > > > > > I updated the KIP to make it explicit that the Java clients shipped > with > > > Apache Kafka 4.0 would only be guaranteed to work with brokers running > > > Apache Kafka 2.1 or newer, since they too would no longer have access > to > > > the removed protocol APIs. Thanks to Jason for asking about this aspect > > > offline. > > > > > > I will leave this KIP open for discussion for another couple or weeks > or > > > so. Please do share your thoughts given the high impact of the change. > > > > > > Ismael > > > > > > On Tue, Jan 3, 2023 at 8:17 AM Ismael Juma wrote: > > > > > > > Hi all, > > > > > > > > I would like to start a discussion regarding the removal of very old > > > > client protocol API versions in Apache Kafka 4.0 to improve > > > maintainability > > > > & supportability of Kafka. Please take a look at the proposal: > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-896%3A+Remove+old+client+protocol+API+versions+in+Kafka+4.0 > > > > > > > > Ismael > > > > > > > > > >
Re: [DISCUSS] KIP-896: Remove old client protocol API versions in Kafka 4.0
Hi Jun, Thanks for the feedback. 1. Good question. I was looking at the IBP issue as more of an inter broker protocol communication thing and hence I thought we'd cover that as part of the KIP that modernizes that aspect (yet to be written). I filed a JIRA recently that shares some of the thinking: https://issues.apache.org/jira/browse/KAFKA-14552. I was hoping we would tackle the details of that as part of the zk deprecation/removal. You're right though that we need to deprecate IBP before 2.1 as part of this KIP since we are removing some of the protocol API versions that would be required for that to work. I'll update the KIP to include this. 2. I believe they are all introduced after 2.1, yes. This is what I used as the source for 2.1: https://github.com/apache/kafka/blob/2.1/clients/src/main/java/org/apache/kafka/common/protocol/ApiKeys.java#L189. Note that I excluded inter broker protocol APIs like LeaderAndIsr, StopReplica and others for the reason I explained in `1`. Does that make sense? Ismael On Mon, Jan 9, 2023 at 11:01 AM Jun Rao wrote: > Hi, Ismael, > > Thanks for the KIP. A couple of comments. > > 1. Should we deprecate IBPs before 2.1 too? > > 2. There seem to be more requests than those listed in the KIP. Are they > all introduced after 2.1? > > Thanks, > > Jun > > > > On Sat, Jan 7, 2023 at 5:13 PM Ismael Juma wrote: > > > Hi all, > > > > I updated the KIP to make it explicit that the Java clients shipped with > > Apache Kafka 4.0 would only be guaranteed to work with brokers running > > Apache Kafka 2.1 or newer, since they too would no longer have access to > > the removed protocol APIs. Thanks to Jason for asking about this aspect > > offline. > > > > I will leave this KIP open for discussion for another couple or weeks or > > so. Please do share your thoughts given the high impact of the change. > > > > Ismael > > > > On Tue, Jan 3, 2023 at 8:17 AM Ismael Juma wrote: > > > > > Hi all, > > > > > > I would like to start a discussion regarding the removal of very old > > > client protocol API versions in Apache Kafka 4.0 to improve > > maintainability > > > & supportability of Kafka. Please take a look at the proposal: > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-896%3A+Remove+old+client+protocol+API+versions+in+Kafka+4.0 > > > > > > Ismael > > > > > >
Re: [DISCUSS] KIP-896: Remove old client protocol API versions in Kafka 4.0
Hi, Ismael, Thanks for the KIP. A couple of comments. 1. Should we deprecate IBPs before 2.1 too? 2. There seem to be more requests than those listed in the KIP. Are they all introduced after 2.1? Thanks, Jun On Sat, Jan 7, 2023 at 5:13 PM Ismael Juma wrote: > Hi all, > > I updated the KIP to make it explicit that the Java clients shipped with > Apache Kafka 4.0 would only be guaranteed to work with brokers running > Apache Kafka 2.1 or newer, since they too would no longer have access to > the removed protocol APIs. Thanks to Jason for asking about this aspect > offline. > > I will leave this KIP open for discussion for another couple or weeks or > so. Please do share your thoughts given the high impact of the change. > > Ismael > > On Tue, Jan 3, 2023 at 8:17 AM Ismael Juma wrote: > > > Hi all, > > > > I would like to start a discussion regarding the removal of very old > > client protocol API versions in Apache Kafka 4.0 to improve > maintainability > > & supportability of Kafka. Please take a look at the proposal: > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-896%3A+Remove+old+client+protocol+API+versions+in+Kafka+4.0 > > > > Ismael > > >
Re: [DISCUSS] KIP-896: Remove old client protocol API versions in Kafka 4.0
Hi all, I updated the KIP to make it explicit that the Java clients shipped with Apache Kafka 4.0 would only be guaranteed to work with brokers running Apache Kafka 2.1 or newer, since they too would no longer have access to the removed protocol APIs. Thanks to Jason for asking about this aspect offline. I will leave this KIP open for discussion for another couple or weeks or so. Please do share your thoughts given the high impact of the change. Ismael On Tue, Jan 3, 2023 at 8:17 AM Ismael Juma wrote: > Hi all, > > I would like to start a discussion regarding the removal of very old > client protocol API versions in Apache Kafka 4.0 to improve maintainability > & supportability of Kafka. Please take a look at the proposal: > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-896%3A+Remove+old+client+protocol+API+versions+in+Kafka+4.0 > > Ismael >
Re: [DISCUSS] KIP-896: Remove old client protocol API versions in Kafka 4.0
Hi Viktor, Thanks for the feedback. See inline. I was gonna ask about whether you think it would be useful to automate > protocol deprecation but I see you'll have a separate KIP for that so I'll > just wait :). > With regards to the follow-up KIP, if you are interested in submitting that, please go ahead. My high level idea was something like the following: Starting with Apache Kafka 5.0, the broker only keeps the latest protocol API version from the first release in previous major version series or a minor release that is at least n years old, whichever is oldest. Reasonable values for `n` could be `3` or `4`. Let's assume `n` is 3 and we release Apache Kafka 5.0 4 years after Apache Kafka 4.0, then we'd keep the highest protocol API version for each protocol from Apache Kafka 4.0. However, if Apache Kafka 5.0 was released 2 years after Apache Kafka 4.0, then we'd pick the relevant 3.x release as the baseline instead. If you do decide to write the KIP, feel free to suggest what you think is best, of course. With that said, let's leave further discussions on this future KIP idea to the future KIP thread. :) I see that in multiple places clients have older than baseline > implementation. Would it make sense in your opinion to also document that > those protocols below the baseline will be deprecated in 4.0 and possibly > go away in next versions (with maybe the deprecation/removal automation) > I considered that, but I thought it would be handled more cleanly by the mechanical policy of the future KIP while this KIP focuses only on what happens in 4.0. Ismael On Wed, Jan 4, 2023 at 7:42 AM Viktor Somogyi-Vass wrote: > Hi Ismael, > > I think this is a good idea, it can also simplify the code quite a bit. > I was gonna ask about whether you think it would be useful to automate > protocol deprecation but I see you'll have a separate KIP for that so I'll > just wait :). > > I see that in multiple places clients have older than baseline > implementation. Would it make sense in your opinion to also document that > those protocols below the baseline will be deprecated in 4.0 and possibly > go away in next versions (with maybe the deprecation/removal automation)? > > Viktor > > On Tue, Jan 3, 2023 at 5:18 PM Ismael Juma wrote: > > > Hi all, > > > > I would like to start a discussion regarding the removal of very old > client > > protocol API versions in Apache Kafka 4.0 to improve maintainability & > > supportability of Kafka. Please take a look at the proposal: > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-896%3A+Remove+old+client+protocol+API+versions+in+Kafka+4.0 > > > > Ismael > > >
Re: [DISCUSS] KIP-896: Remove old client protocol API versions in Kafka 4.0
Hi Ismael, I think this is a good idea, it can also simplify the code quite a bit. I was gonna ask about whether you think it would be useful to automate protocol deprecation but I see you'll have a separate KIP for that so I'll just wait :). I see that in multiple places clients have older than baseline implementation. Would it make sense in your opinion to also document that those protocols below the baseline will be deprecated in 4.0 and possibly go away in next versions (with maybe the deprecation/removal automation)? Viktor On Tue, Jan 3, 2023 at 5:18 PM Ismael Juma wrote: > Hi all, > > I would like to start a discussion regarding the removal of very old client > protocol API versions in Apache Kafka 4.0 to improve maintainability & > supportability of Kafka. Please take a look at the proposal: > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-896%3A+Remove+old+client+protocol+API+versions+in+Kafka+4.0 > > Ismael >
