Re: [DISCUSS] KIP-915: Next Gen Group Coordinator Downgrade Path

Thanks, Kim. The KIP looks good to me now. Jun On Fri, Mar 31, 2023 at 10:00 AM Jeff Kim wrote: > Hi Jun, > > Thanks for the comment. I have updated the statement: > "Note that introducing a new non-tagged field or removing an > existing non-tagged field in the future will not be backward compa

Re: [DISCUSS] KIP-915: Next Gen Group Coordinator Downgrade Path

Hi Jun, Thanks for the comment. I have updated the statement: "Note that introducing a new non-tagged field or removing an existing non-tagged field in the future will not be backward compatible." Best, Jeff On Fri, Mar 31, 2023 at 12:57 PM Jun Rao wrote: > Thanks for the reply Jeff. > > 2. Co

Re: [DISCUSS] KIP-915: Next Gen Group Coordinator Downgrade Path

Thanks for the reply Jeff. 2. Could we document that removing a non-tagged field is not backward compatible in the KIP? Jun On Fri, Mar 31, 2023 at 7:17 AM Jeff Kim wrote: > Hi Jun, > > Thanks for the response. > > 1. Thanks for the catch. I updated the comment to say > "// KIP-915 bumping the

Re: [DISCUSS] KIP-915: Next Gen Group Coordinator Downgrade Path

Hi Jun, Thanks for the response. 1. Thanks for the catch. I updated the comment to say "// KIP-915 bumping the version will no longer make this record backward compatible." 2. Yes. Adding/removing a non-tagged field is not backward compatible. We will not enforce this but instead let the develop

Re: [DISCUSS] KIP-915: Next Gen Group Coordinator Downgrade Path

Hi, Jeff, Thanks for the reply. 1. There are 3 references of "// KIP-915 do not bump the version" in green in the KIP. Are those expected? 2. So, does that mean downgrade doesn't support the removal of a non-tagged field? Thanks, Jun On Thu, Mar 30, 2023 at 2:57 PM Jeff Kim wrote: > Hi Isma

Re: [DISCUSS] KIP-915: Next Gen Group Coordinator Downgrade Path

Hi Ismael and Jun, Thank you for the comments. Ismael, I updated the KIP with "note: we will need commitment from release managers to perform the minor releases". Let me know your thoughts. Jun, 1. I am unsure where the KIP states that we will not bump the version of TransactionLogValue. The K

Re: [DISCUSS] KIP-915: Next Gen Group Coordinator Downgrade Path

Hi, Jeff, Thanks for the KIP. A couple of comments. 1. The comment says that KIP-915 does not bump the version of TransactionLogValue.json, but the schema seems to bump the version from 0 to 1. 2. How do we support downgrade when a field is removed? Does it require the removed field to have a d

Re: [DISCUSS] KIP-915: Next Gen Group Coordinator Downgrade Path

Hi, I'm late to this conversation, but it's a bit odd to backport the flexible versions change to "3.0.3, 3.1.3, 3.2.4, 3.3.3, and 3.4.1" unless we intend to release an update for each of these. So, I suggest that either we commit to releasing an update for each of these versions or we reduce the

Re: [DISCUSS] KIP-915: Next Gen Group Coordinator Downgrade Path

Hi Justine, I think that unknown tagged fields are just ignored. This means that they will be set to their default values. We would have to rely on the default to determine if we can use them or not. For instance, we could use a sentinel value (e.g -1 or null) as default. David On Wed, Mar 29, 2

Re: [DISCUSS] KIP-915: Next Gen Group Coordinator Downgrade Path

Hey Jeff, Thanks for the response. I think I was wondering about the reverse scenario. If the coordinator supported field x, but it wasn't present in the record. In that case, will we have some sort of handling to not use that field? Justine On Wed, Mar 29, 2023 at 2:31 PM Jeff Kim wrote: > Hi

Re: [DISCUSS] KIP-915: Next Gen Group Coordinator Downgrade Path

Hi Justine, Thank you for the comment. Yes, that is what we do today. Let's say there's a record that has two tagged fields, x and y. If the coordinator's schema only supports field x, that will be the only field it reads. Thanks, Jeff On Wed, Mar 29, 2023 at 12:43 PM Justine Olshan wrote: > H

Re: [DISCUSS] KIP-915: Next Gen Group Coordinator Downgrade Path

Hey Jeff, Going off of Alexandre's point -- if we have a tagged field in one version but not another (or even in the same version we could have one with and one without) -- are we simply checking if the tagged field exists in the message before we try to read it? I think this is the sort of thing

Re: [DISCUSS] KIP-915: Next Gen Group Coordinator Downgrade Path

Hi Alexandre, Thanks for the feedback. 100. You're correct. I updated the KIP to explicitly state that once we introduce a new tagged field in a later version, that version can never be downgraded to the listed versions. 101. I don't think there is a way to differentiate the two but also not sur

Re: [DISCUSS] KIP-915: Next Gen Group Coordinator Downgrade Path

Hi Jeff, Thank you for the fast answer! 100. Got it, I think I am starting to understand based on your example exposing three Kafka versions. Please let me know if the following corresponds to the correct interpretation. Taking OffsetCommitValue as an example, the idea is to bump the schema to v

Re: [DISCUSS] KIP-915: Next Gen Group Coordinator Downgrade Path

Hi Alexandre, Thank you for taking a look! 100. I am not sure I fully understand what you mean by forcefully adding tagged fields. Let's say VX does not have a flexible version, VY allows deserialization but serializes with a non-flexible version, and VZ introduces a new tagged field. VX upgrade

Re: [DISCUSS] KIP-915: Next Gen Group Coordinator Downgrade Path

Hi Jeff, Team, Thank you for the KIP. This is a very interesting approach. I feel it is simpler than the described alternative although it comes with tradeoffs, thanks for highlighting those. If I may, I would like to share two naive questions. 100. The KIP mentions that records will be serialise

Re: [DISCUSS] KIP-915: Next Gen Group Coordinator Downgrade Path

Hi Yi, > Does it mean with a flexible version, the future version of these value types will stay at the version where the flexibility is first introduced? Will there be any need to bump the version again in the future? Yes, there will be no need to bump the version since we will only be adding ta

Re: [DISCUSS] KIP-915: Next Gen Group Coordinator Downgrade Path

Hi Jeff, Thanks for the update. Does it mean with a flexible version, the future version of these value types will stay at the version where the flexibility is first introduced? Will there be any need to bump the version again in the future? To enforce the version not bumping, is it possible to ha

Re: [DISCUSS] KIP-915: Next Gen Group Coordinator Downgrade Path

Hi all, After discussing with my colleagues, I have repurposed the KIP for a general downgrade solution for both transaction and group coordinators. The KIP no longer discusses the downgrade path but instead lays out the foundation for future downgrade solutions. Link: https://cwiki.apache.org/co

Re: [DISCUSS] KIP-915: Next Gen Group Coordinator Downgrade Path

Hi David and Justine, Thank you both for the detailed feedback. David, 1. That makes sense. I revised the "Reading new fields" section with how we can downgrade to the highest known version and that this was confirmed via unit testing. I also attempted to dive deeper into using tagged fields and

Re: [DISCUSS] KIP-915: Next Gen Group Coordinator Downgrade Path

Hey Jeff and David, Thanks for the KIP! I was also looking into this a bit since I may want to change the record format for KIP-890 as well (finally implementing the record change from KIP-360 to better support the epoch bump. This will potentially be helpful for me to implement that work. I disc

Re: [DISCUSS] KIP-915: Next Gen Group Coordinator Downgrade Path

Hi Jeff, Thanks for the KIP! I am really glad that we are finally addressing this gap in KIP-848. I have a few general comments. 1. Overall, I feel like the important bits are not bold enough in the KIP. I think that it is good to explain the overall upgrade/downgrade process and to highlight whe

[DISCUSS] KIP-915: Next Gen Group Coordinator Downgrade Path

Hi folks, I would like to start a discussion thread for KIP-915: Next Gen Group Coordinator Downgrade Path which proposes the downgrade design for the new group coordinator introduced in KIP-848