Re: [DISCUSS] KIP-915: Next Gen Group Coordinator Downgrade Path

[Jun Rao]

Thanks, Kim. The KIP looks good to me now.

Jun

On Fri, Mar 31, 2023 at 10:00 AM Jeff Kim 
wrote:

> Hi Jun,
>
> Thanks for the comment. I have updated the statement:
> "Note that introducing a new non-tagged field or removing an
> existing non-tagged field in the future will not be backward compatible."
>
> Best,
> Jeff
>
> On Fri, Mar 31, 2023 at 12:57 PM Jun Rao  wrote:
>
> > Thanks for the reply Jeff.
> >
> > 2. Could we document that removing a non-tagged field is not backward
> > compatible in the KIP?
> >
> > Jun
> >
> > On Fri, Mar 31, 2023 at 7:17 AM Jeff Kim 
> > wrote:
> >
> > > Hi Jun,
> > >
> > > Thanks for the response.
> > >
> > > 1. Thanks for the catch. I updated the comment to say
> > > "// KIP-915 bumping the version will no longer make this
> > > record backward compatible."
> > >
> > > 2. Yes. Adding/removing a non-tagged field is not backward compatible.
> > > We will not enforce this but instead let the developer know through
> > > the comment. This pushes us to use tagged fields but gives us the
> > > flexibility to introduce non backward compatible change.
> > >
> > > Best,
> > > Jeff
> > >
> > >
> > > On Thu, Mar 30, 2023 at 7:06 PM Jun Rao 
> > wrote:
> > >
> > > > Hi, Jeff,
> > > >
> > > > Thanks for the reply.
> > > >
> > > > 1. There are 3 references of "// KIP-915 do not bump the version" in
> > > green
> > > > in the KIP. Are those expected?
> > > >
> > > > 2. So, does that mean downgrade doesn't support the removal of a
> > > non-tagged
> > > > field?
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Thu, Mar 30, 2023 at 2:57 PM Jeff Kim
>  > >
> > > > wrote:
> > > >
> > > > > Hi Ismael and Jun,
> > > > >
> > > > > Thank you for the comments.
> > > > >
> > > > > Ismael,
> > > > >
> > > > > I updated the KIP with "note: we will need commitment from
> > > > > release managers to perform the minor releases".
> > > > > Let me know your thoughts.
> > > > >
> > > > > Jun,
> > > > >
> > > > > 1. I am unsure where the KIP states that we will not bump the
> version
> > > > > of TransactionLogValue. The KIP proposes to bump all Value
> > > > > record types, so that statement should be removed.
> > > > >
> > > > > 2. Deserialization starts by iterating through the list of tagged
> > > fields
> > > > > then identifying ones that the schema knows about. So, even if the
> > > > > downgraded
> > > > > schema expects a tagged field, if it doesn't exist in the record
> > > > > (removed in later version) then it is ignored.
> > > > >
> > > > > Best,
> > > > > Jeff
> > > > >
> > > > > On Thu, Mar 30, 2023 at 5:16 PM Jun Rao 
> > > > wrote:
> > > > >
> > > > > > Hi, Jeff,
> > > > > >
> > > > > > Thanks for the KIP. A couple of comments.
> > > > > >
> > > > > > 1. The comment says that  KIP-915 does not bump the version of
> > > > > > TransactionLogValue.json, but the schema seems to bump the
> version
> > > > from 0
> > > > > > to 1.
> > > > > >
> > > > > > 2. How do we support downgrade when a field is removed? Does it
> > > require
> > > > > the
> > > > > > removed field to have a default?
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > >
> > > > > > On Thu, Mar 30, 2023 at 6:49 AM Ismael Juma 
> > > wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > I'm late to this conversation, but it's a bit odd to backport
> the
> > > > > > flexible
> > > > > > > versions change to "3.0.3, 3.1.3, 3.2.4, 3.3.3, and 3.4.1"
> unless
> > > we
> > > > > > intend
> > > > > > > to release an update for each of these. So, I suggest that
> either
> > > we
> > > > > > commit
> > > > > > > to releasing an update for each of these versions or we reduce
> > the
> > > > set
> > > > > of
> > > > > > > versions we backport the change to.
> > > > > > >
> > > > > > > Ismael
> > > > > > >
> > > > > > > On Wed, Mar 15, 2023 at 4:57 PM Jeff Kim
> > > >  > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi folks,
> > > > > > > >
> > > > > > > > I would like to start a discussion thread for KIP-915: Next
> Gen
> > > > Group
> > > > > > > > Coordinator Downgrade Path which proposes the downgrade
> design
> > > for
> > > > > the
> > > > > > > new
> > > > > > > > group coordinator introduced in KIP-848
> > > > > > > > <
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-848%3A+The+Next+Generation+of+the+Consumer+Rebalance+Protocol
> > > > > > > > >
> > > > > > > > .
> > > > > > > >
> > > > > > > > KIP:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-915%3A+Next+Gen+Group+Coordinator+Downgrade+Path
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Jeff
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-915: Next Gen Group Coordinator Downgrade Path

[Jeff Kim]

Hi Jun,

Thanks for the comment. I have updated the statement:
"Note that introducing a new non-tagged field or removing an
existing non-tagged field in the future will not be backward compatible."

Best,
Jeff

On Fri, Mar 31, 2023 at 12:57 PM Jun Rao  wrote:

> Thanks for the reply Jeff.
>
> 2. Could we document that removing a non-tagged field is not backward
> compatible in the KIP?
>
> Jun
>
> On Fri, Mar 31, 2023 at 7:17 AM Jeff Kim 
> wrote:
>
> > Hi Jun,
> >
> > Thanks for the response.
> >
> > 1. Thanks for the catch. I updated the comment to say
> > "// KIP-915 bumping the version will no longer make this
> > record backward compatible."
> >
> > 2. Yes. Adding/removing a non-tagged field is not backward compatible.
> > We will not enforce this but instead let the developer know through
> > the comment. This pushes us to use tagged fields but gives us the
> > flexibility to introduce non backward compatible change.
> >
> > Best,
> > Jeff
> >
> >
> > On Thu, Mar 30, 2023 at 7:06 PM Jun Rao 
> wrote:
> >
> > > Hi, Jeff,
> > >
> > > Thanks for the reply.
> > >
> > > 1. There are 3 references of "// KIP-915 do not bump the version" in
> > green
> > > in the KIP. Are those expected?
> > >
> > > 2. So, does that mean downgrade doesn't support the removal of a
> > non-tagged
> > > field?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Thu, Mar 30, 2023 at 2:57 PM Jeff Kim  >
> > > wrote:
> > >
> > > > Hi Ismael and Jun,
> > > >
> > > > Thank you for the comments.
> > > >
> > > > Ismael,
> > > >
> > > > I updated the KIP with "note: we will need commitment from
> > > > release managers to perform the minor releases".
> > > > Let me know your thoughts.
> > > >
> > > > Jun,
> > > >
> > > > 1. I am unsure where the KIP states that we will not bump the version
> > > > of TransactionLogValue. The KIP proposes to bump all Value
> > > > record types, so that statement should be removed.
> > > >
> > > > 2. Deserialization starts by iterating through the list of tagged
> > fields
> > > > then identifying ones that the schema knows about. So, even if the
> > > > downgraded
> > > > schema expects a tagged field, if it doesn't exist in the record
> > > > (removed in later version) then it is ignored.
> > > >
> > > > Best,
> > > > Jeff
> > > >
> > > > On Thu, Mar 30, 2023 at 5:16 PM Jun Rao 
> > > wrote:
> > > >
> > > > > Hi, Jeff,
> > > > >
> > > > > Thanks for the KIP. A couple of comments.
> > > > >
> > > > > 1. The comment says that  KIP-915 does not bump the version of
> > > > > TransactionLogValue.json, but the schema seems to bump the version
> > > from 0
> > > > > to 1.
> > > > >
> > > > > 2. How do we support downgrade when a field is removed? Does it
> > require
> > > > the
> > > > > removed field to have a default?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > >
> > > > > On Thu, Mar 30, 2023 at 6:49 AM Ismael Juma 
> > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I'm late to this conversation, but it's a bit odd to backport the
> > > > > flexible
> > > > > > versions change to "3.0.3, 3.1.3, 3.2.4, 3.3.3, and 3.4.1" unless
> > we
> > > > > intend
> > > > > > to release an update for each of these. So, I suggest that either
> > we
> > > > > commit
> > > > > > to releasing an update for each of these versions or we reduce
> the
> > > set
> > > > of
> > > > > > versions we backport the change to.
> > > > > >
> > > > > > Ismael
> > > > > >
> > > > > > On Wed, Mar 15, 2023 at 4:57 PM Jeff Kim
> > >  > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi folks,
> > > > > > >
> > > > > > > I would like to start a discussion thread for KIP-915: Next Gen
> > > Group
> > > > > > > Coordinator Downgrade Path which proposes the downgrade design
> > for
> > > > the
> > > > > > new
> > > > > > > group coordinator introduced in KIP-848
> > > > > > > <
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-848%3A+The+Next+Generation+of+the+Consumer+Rebalance+Protocol
> > > > > > > >
> > > > > > > .
> > > > > > >
> > > > > > > KIP:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-915%3A+Next+Gen+Group+Coordinator+Downgrade+Path
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Jeff
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-915: Next Gen Group Coordinator Downgrade Path

[Jun Rao]

Thanks for the reply Jeff.

2. Could we document that removing a non-tagged field is not backward
compatible in the KIP?

Jun

On Fri, Mar 31, 2023 at 7:17 AM Jeff Kim 
wrote:

> Hi Jun,
>
> Thanks for the response.
>
> 1. Thanks for the catch. I updated the comment to say
> "// KIP-915 bumping the version will no longer make this
> record backward compatible."
>
> 2. Yes. Adding/removing a non-tagged field is not backward compatible.
> We will not enforce this but instead let the developer know through
> the comment. This pushes us to use tagged fields but gives us the
> flexibility to introduce non backward compatible change.
>
> Best,
> Jeff
>
>
> On Thu, Mar 30, 2023 at 7:06 PM Jun Rao  wrote:
>
> > Hi, Jeff,
> >
> > Thanks for the reply.
> >
> > 1. There are 3 references of "// KIP-915 do not bump the version" in
> green
> > in the KIP. Are those expected?
> >
> > 2. So, does that mean downgrade doesn't support the removal of a
> non-tagged
> > field?
> >
> > Thanks,
> >
> > Jun
> >
> > On Thu, Mar 30, 2023 at 2:57 PM Jeff Kim 
> > wrote:
> >
> > > Hi Ismael and Jun,
> > >
> > > Thank you for the comments.
> > >
> > > Ismael,
> > >
> > > I updated the KIP with "note: we will need commitment from
> > > release managers to perform the minor releases".
> > > Let me know your thoughts.
> > >
> > > Jun,
> > >
> > > 1. I am unsure where the KIP states that we will not bump the version
> > > of TransactionLogValue. The KIP proposes to bump all Value
> > > record types, so that statement should be removed.
> > >
> > > 2. Deserialization starts by iterating through the list of tagged
> fields
> > > then identifying ones that the schema knows about. So, even if the
> > > downgraded
> > > schema expects a tagged field, if it doesn't exist in the record
> > > (removed in later version) then it is ignored.
> > >
> > > Best,
> > > Jeff
> > >
> > > On Thu, Mar 30, 2023 at 5:16 PM Jun Rao 
> > wrote:
> > >
> > > > Hi, Jeff,
> > > >
> > > > Thanks for the KIP. A couple of comments.
> > > >
> > > > 1. The comment says that  KIP-915 does not bump the version of
> > > > TransactionLogValue.json, but the schema seems to bump the version
> > from 0
> > > > to 1.
> > > >
> > > > 2. How do we support downgrade when a field is removed? Does it
> require
> > > the
> > > > removed field to have a default?
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > >
> > > > On Thu, Mar 30, 2023 at 6:49 AM Ismael Juma 
> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I'm late to this conversation, but it's a bit odd to backport the
> > > > flexible
> > > > > versions change to "3.0.3, 3.1.3, 3.2.4, 3.3.3, and 3.4.1" unless
> we
> > > > intend
> > > > > to release an update for each of these. So, I suggest that either
> we
> > > > commit
> > > > > to releasing an update for each of these versions or we reduce the
> > set
> > > of
> > > > > versions we backport the change to.
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Wed, Mar 15, 2023 at 4:57 PM Jeff Kim
> >  > > >
> > > > > wrote:
> > > > >
> > > > > > Hi folks,
> > > > > >
> > > > > > I would like to start a discussion thread for KIP-915: Next Gen
> > Group
> > > > > > Coordinator Downgrade Path which proposes the downgrade design
> for
> > > the
> > > > > new
> > > > > > group coordinator introduced in KIP-848
> > > > > > <
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-848%3A+The+Next+Generation+of+the+Consumer+Rebalance+Protocol
> > > > > > >
> > > > > > .
> > > > > >
> > > > > > KIP:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-915%3A+Next+Gen+Group+Coordinator+Downgrade+Path
> > > > > >
> > > > > > Thanks,
> > > > > > Jeff
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-915: Next Gen Group Coordinator Downgrade Path

[Jeff Kim]

Hi Jun,

Thanks for the response.

1. Thanks for the catch. I updated the comment to say
"// KIP-915 bumping the version will no longer make this
record backward compatible."

2. Yes. Adding/removing a non-tagged field is not backward compatible.
We will not enforce this but instead let the developer know through
the comment. This pushes us to use tagged fields but gives us the
flexibility to introduce non backward compatible change.

Best,
Jeff


On Thu, Mar 30, 2023 at 7:06 PM Jun Rao  wrote:

> Hi, Jeff,
>
> Thanks for the reply.
>
> 1. There are 3 references of "// KIP-915 do not bump the version" in green
> in the KIP. Are those expected?
>
> 2. So, does that mean downgrade doesn't support the removal of a non-tagged
> field?
>
> Thanks,
>
> Jun
>
> On Thu, Mar 30, 2023 at 2:57 PM Jeff Kim 
> wrote:
>
> > Hi Ismael and Jun,
> >
> > Thank you for the comments.
> >
> > Ismael,
> >
> > I updated the KIP with "note: we will need commitment from
> > release managers to perform the minor releases".
> > Let me know your thoughts.
> >
> > Jun,
> >
> > 1. I am unsure where the KIP states that we will not bump the version
> > of TransactionLogValue. The KIP proposes to bump all Value
> > record types, so that statement should be removed.
> >
> > 2. Deserialization starts by iterating through the list of tagged fields
> > then identifying ones that the schema knows about. So, even if the
> > downgraded
> > schema expects a tagged field, if it doesn't exist in the record
> > (removed in later version) then it is ignored.
> >
> > Best,
> > Jeff
> >
> > On Thu, Mar 30, 2023 at 5:16 PM Jun Rao 
> wrote:
> >
> > > Hi, Jeff,
> > >
> > > Thanks for the KIP. A couple of comments.
> > >
> > > 1. The comment says that  KIP-915 does not bump the version of
> > > TransactionLogValue.json, but the schema seems to bump the version
> from 0
> > > to 1.
> > >
> > > 2. How do we support downgrade when a field is removed? Does it require
> > the
> > > removed field to have a default?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Thu, Mar 30, 2023 at 6:49 AM Ismael Juma  wrote:
> > >
> > > > Hi,
> > > >
> > > > I'm late to this conversation, but it's a bit odd to backport the
> > > flexible
> > > > versions change to "3.0.3, 3.1.3, 3.2.4, 3.3.3, and 3.4.1" unless we
> > > intend
> > > > to release an update for each of these. So, I suggest that either we
> > > commit
> > > > to releasing an update for each of these versions or we reduce the
> set
> > of
> > > > versions we backport the change to.
> > > >
> > > > Ismael
> > > >
> > > > On Wed, Mar 15, 2023 at 4:57 PM Jeff Kim
>  > >
> > > > wrote:
> > > >
> > > > > Hi folks,
> > > > >
> > > > > I would like to start a discussion thread for KIP-915: Next Gen
> Group
> > > > > Coordinator Downgrade Path which proposes the downgrade design for
> > the
> > > > new
> > > > > group coordinator introduced in KIP-848
> > > > > <
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-848%3A+The+Next+Generation+of+the+Consumer+Rebalance+Protocol
> > > > > >
> > > > > .
> > > > >
> > > > > KIP:
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-915%3A+Next+Gen+Group+Coordinator+Downgrade+Path
> > > > >
> > > > > Thanks,
> > > > > Jeff
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-915: Next Gen Group Coordinator Downgrade Path

[Jun Rao]

Hi, Jeff,

Thanks for the reply.

1. There are 3 references of "// KIP-915 do not bump the version" in green
in the KIP. Are those expected?

2. So, does that mean downgrade doesn't support the removal of a non-tagged
field?

Thanks,

Jun

On Thu, Mar 30, 2023 at 2:57 PM Jeff Kim 
wrote:

> Hi Ismael and Jun,
>
> Thank you for the comments.
>
> Ismael,
>
> I updated the KIP with "note: we will need commitment from
> release managers to perform the minor releases".
> Let me know your thoughts.
>
> Jun,
>
> 1. I am unsure where the KIP states that we will not bump the version
> of TransactionLogValue. The KIP proposes to bump all Value
> record types, so that statement should be removed.
>
> 2. Deserialization starts by iterating through the list of tagged fields
> then identifying ones that the schema knows about. So, even if the
> downgraded
> schema expects a tagged field, if it doesn't exist in the record
> (removed in later version) then it is ignored.
>
> Best,
> Jeff
>
> On Thu, Mar 30, 2023 at 5:16 PM Jun Rao  wrote:
>
> > Hi, Jeff,
> >
> > Thanks for the KIP. A couple of comments.
> >
> > 1. The comment says that  KIP-915 does not bump the version of
> > TransactionLogValue.json, but the schema seems to bump the version from 0
> > to 1.
> >
> > 2. How do we support downgrade when a field is removed? Does it require
> the
> > removed field to have a default?
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Thu, Mar 30, 2023 at 6:49 AM Ismael Juma  wrote:
> >
> > > Hi,
> > >
> > > I'm late to this conversation, but it's a bit odd to backport the
> > flexible
> > > versions change to "3.0.3, 3.1.3, 3.2.4, 3.3.3, and 3.4.1" unless we
> > intend
> > > to release an update for each of these. So, I suggest that either we
> > commit
> > > to releasing an update for each of these versions or we reduce the set
> of
> > > versions we backport the change to.
> > >
> > > Ismael
> > >
> > > On Wed, Mar 15, 2023 at 4:57 PM Jeff Kim  >
> > > wrote:
> > >
> > > > Hi folks,
> > > >
> > > > I would like to start a discussion thread for KIP-915: Next Gen Group
> > > > Coordinator Downgrade Path which proposes the downgrade design for
> the
> > > new
> > > > group coordinator introduced in KIP-848
> > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-848%3A+The+Next+Generation+of+the+Consumer+Rebalance+Protocol
> > > > >
> > > > .
> > > >
> > > > KIP:
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-915%3A+Next+Gen+Group+Coordinator+Downgrade+Path
> > > >
> > > > Thanks,
> > > > Jeff
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-915: Next Gen Group Coordinator Downgrade Path

[Jeff Kim]

Hi Ismael and Jun,

Thank you for the comments.

Ismael,

I updated the KIP with "note: we will need commitment from
release managers to perform the minor releases".
Let me know your thoughts.

Jun,

1. I am unsure where the KIP states that we will not bump the version
of TransactionLogValue. The KIP proposes to bump all Value
record types, so that statement should be removed.

2. Deserialization starts by iterating through the list of tagged fields
then identifying ones that the schema knows about. So, even if the
downgraded
schema expects a tagged field, if it doesn't exist in the record
(removed in later version) then it is ignored.

Best,
Jeff

On Thu, Mar 30, 2023 at 5:16 PM Jun Rao  wrote:

> Hi, Jeff,
>
> Thanks for the KIP. A couple of comments.
>
> 1. The comment says that  KIP-915 does not bump the version of
> TransactionLogValue.json, but the schema seems to bump the version from 0
> to 1.
>
> 2. How do we support downgrade when a field is removed? Does it require the
> removed field to have a default?
>
> Thanks,
>
> Jun
>
>
> On Thu, Mar 30, 2023 at 6:49 AM Ismael Juma  wrote:
>
> > Hi,
> >
> > I'm late to this conversation, but it's a bit odd to backport the
> flexible
> > versions change to "3.0.3, 3.1.3, 3.2.4, 3.3.3, and 3.4.1" unless we
> intend
> > to release an update for each of these. So, I suggest that either we
> commit
> > to releasing an update for each of these versions or we reduce the set of
> > versions we backport the change to.
> >
> > Ismael
> >
> > On Wed, Mar 15, 2023 at 4:57 PM Jeff Kim 
> > wrote:
> >
> > > Hi folks,
> > >
> > > I would like to start a discussion thread for KIP-915: Next Gen Group
> > > Coordinator Downgrade Path which proposes the downgrade design for the
> > new
> > > group coordinator introduced in KIP-848
> > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-848%3A+The+Next+Generation+of+the+Consumer+Rebalance+Protocol
> > > >
> > > .
> > >
> > > KIP:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-915%3A+Next+Gen+Group+Coordinator+Downgrade+Path
> > >
> > > Thanks,
> > > Jeff
> > >
> >
>

Re: [DISCUSS] KIP-915: Next Gen Group Coordinator Downgrade Path

[Jun Rao]

Hi, Jeff,

Thanks for the KIP. A couple of comments.

1. The comment says that  KIP-915 does not bump the version of
TransactionLogValue.json, but the schema seems to bump the version from 0
to 1.

2. How do we support downgrade when a field is removed? Does it require the
removed field to have a default?

Thanks,

Jun


On Thu, Mar 30, 2023 at 6:49 AM Ismael Juma  wrote:

> Hi,
>
> I'm late to this conversation, but it's a bit odd to backport the flexible
> versions change to "3.0.3, 3.1.3, 3.2.4, 3.3.3, and 3.4.1" unless we intend
> to release an update for each of these. So, I suggest that either we commit
> to releasing an update for each of these versions or we reduce the set of
> versions we backport the change to.
>
> Ismael
>
> On Wed, Mar 15, 2023 at 4:57 PM Jeff Kim 
> wrote:
>
> > Hi folks,
> >
> > I would like to start a discussion thread for KIP-915: Next Gen Group
> > Coordinator Downgrade Path which proposes the downgrade design for the
> new
> > group coordinator introduced in KIP-848
> > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-848%3A+The+Next+Generation+of+the+Consumer+Rebalance+Protocol
> > >
> > .
> >
> > KIP:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-915%3A+Next+Gen+Group+Coordinator+Downgrade+Path
> >
> > Thanks,
> > Jeff
> >
>

Re: [DISCUSS] KIP-915: Next Gen Group Coordinator Downgrade Path

[Ismael Juma]

Hi,

I'm late to this conversation, but it's a bit odd to backport the flexible
versions change to "3.0.3, 3.1.3, 3.2.4, 3.3.3, and 3.4.1" unless we intend
to release an update for each of these. So, I suggest that either we commit
to releasing an update for each of these versions or we reduce the set of
versions we backport the change to.

Ismael

On Wed, Mar 15, 2023 at 4:57 PM Jeff Kim 
wrote:

> Hi folks,
>
> I would like to start a discussion thread for KIP-915: Next Gen Group
> Coordinator Downgrade Path which proposes the downgrade design for the new
> group coordinator introduced in KIP-848
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-848%3A+The+Next+Generation+of+the+Consumer+Rebalance+Protocol
> >
> .
>
> KIP:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-915%3A+Next+Gen+Group+Coordinator+Downgrade+Path
>
> Thanks,
> Jeff
>

Re: [DISCUSS] KIP-915: Next Gen Group Coordinator Downgrade Path

[David Jacot]

Hi Justine,

I think that unknown tagged fields are just ignored. This means that they
will be set to their default values. We would have to rely on the default
to determine if we can use them or not. For instance, we could use a
sentinel value (e.g -1 or null) as default.

David

On Wed, Mar 29, 2023 at 11:56 PM Justine Olshan
 wrote:

> Hey Jeff,
>
> Thanks for the response. I think I was wondering about the reverse
> scenario. If the coordinator supported field x, but it wasn't present in
> the record.
> In that case, will we have some sort of handling to not use that field?
>
> Justine
>
> On Wed, Mar 29, 2023 at 2:31 PM Jeff Kim 
> wrote:
>
> > Hi Justine,
> >
> > Thank you for the comment. Yes, that is what we do today. Let's say
> there's
> > a record
> > that has two tagged fields, x and y. If the coordinator's schema only
> > supports field x, that will be the only field it reads.
> >
> > Thanks,
> > Jeff
> >
> > On Wed, Mar 29, 2023 at 12:43 PM Justine Olshan
> >  wrote:
> >
> > > Hey Jeff,
> > >
> > > Going off of Alexandre's point -- if we have a tagged field in one
> > version
> > > but not another (or even in the same version we could have one with and
> > one
> > > without) -- are we simply checking if the tagged field exists in the
> > > message before we try to read it?
> > > I think this is the sort of thing we already do, but just wanted to
> > > clarify. We don't need to distinguish versions since we just need to
> > > distinguish if the tagged field is there or not.
> > >
> > > Thanks,
> > > Justine
> > >
> > >
> > > On Tue, Mar 28, 2023 at 3:42 PM Jeff Kim  >
> > > wrote:
> > >
> > > > Hi Alexandre,
> > > >
> > > > Thanks for the feedback.
> > > >
> > > > 100. You're correct. I updated the KIP to explicitly state that once
> we
> > > > introduce a new tagged field in a later version, that version can
> never
> > > be
> > > > downgraded to the listed versions.
> > > >
> > > > 101. I don't think there is a way to differentiate the two but also
> not
> > > > sure if there will be a need to.
> > > >
> > > > Best,
> > > > Jeff
> > > >
> > > > On Tue, Mar 28, 2023 at 2:13 PM Alexandre Dupriez <
> > > > alexandre.dupr...@gmail.com> wrote:
> > > >
> > > > > Hi Jeff,
> > > > >
> > > > > Thank you for the fast answer!
> > > > >
> > > > > 100. Got it, I think I am starting to understand based on your
> > example
> > > > > exposing three Kafka versions. Please let me know if the following
> > > > > corresponds to the correct interpretation.
> > > > >
> > > > > Taking OffsetCommitValue as an example, the idea is to bump the
> > schema
> > > > > to version 4 to make the record flexible. The schema version 4 will
> > be
> > > > > back-ported to versions 3.0.3, 3.1.3, 3.2.4, 3.3.3, 3.4.1 and
> > > > > potentially 3.5. These versions will continue to serialise the
> record
> > > > > with version 3 so that downgrades from there to earlier minor
> and/or
> > > > > major versions are possible. And, these versions will deserialize
> the
> > > > > records with version 4 so that they can support flexible records,
> > > > > although they do not make use of tagged fields (if any is present).
> > > > >
> > > > > Then, in a future version, 3.x, x >= 6, a tag field will be added
> to
> > > > > the record - say, topic id. Downgrade from 3.x to 3.0.3, 3.1.3,
> > 3.2.4,
> > > > > 3.3.3, 3.4.1 and 3.5 will be possible because these versions will
> > > > > perform deserialisation with the record schema version 4, that is,
> > > > > supporting tagged fields.
> > > > >
> > > > > Hence, this approach is both backward and forward looking and
> allows
> > > > > to extend the scope of compatible versions for downgrades.
> > > > >
> > > > > [N] 3.x, x >= 6 --> downgradable to [I] 3.0.3, 3.1.3, 3.2.4, 3.3.3,
> > > > > 3.4.1, 3.5 --> downgradable to e.g. [O] 3.0.0.
> > > > >
> > > > > One note though is that a transitive downgrade from 3.x, x >= 6, to
> > > > > 3.0.0 via the version domain [I], will not be supported. Should it
> be
> > > > > explicitly mentioned in the KIP that downgrades from 3.0.3, 3.1.3,
> > > > > 3.2.4, 3.3.3, 3.4.1, 3.5 to earlier versions may not be possible
> (if
> > > > > offsets or group metadata records were generated by a higher
> version
> > > > > 3.x, x >= 6)? Or am I still misunderstanding?
> > > > >
> > > > > 101. I will try to provide an example to illustrate what I mean by
> > > > > version. Consider the GroupMetadataValue schema version 4 and the
> > > > > tagged field topicId introduced in 3.6. Let's say a new optional
> > field
> > > > > needs to be added in 3.7. We will have two records version 4, one
> > with
> > > > > topic id, the other with topic id and the new field. The new field
> is
> > > > > semantically optional (structurally it is always optional since it
> is
> > > > > a tagged field) but we want to make the distinction between a
> record
> > > > > generated by 3.6 and one generated by 3.7. How do we resolve the
> > > > > ambiguity?
> > > > >
> > > > > Thanks!
> > > > > 

Re: [DISCUSS] KIP-915: Next Gen Group Coordinator Downgrade Path

[Justine Olshan]

Hey Jeff,

Thanks for the response. I think I was wondering about the reverse
scenario. If the coordinator supported field x, but it wasn't present in
the record.
In that case, will we have some sort of handling to not use that field?

Justine

On Wed, Mar 29, 2023 at 2:31 PM Jeff Kim 
wrote:

> Hi Justine,
>
> Thank you for the comment. Yes, that is what we do today. Let's say there's
> a record
> that has two tagged fields, x and y. If the coordinator's schema only
> supports field x, that will be the only field it reads.
>
> Thanks,
> Jeff
>
> On Wed, Mar 29, 2023 at 12:43 PM Justine Olshan
>  wrote:
>
> > Hey Jeff,
> >
> > Going off of Alexandre's point -- if we have a tagged field in one
> version
> > but not another (or even in the same version we could have one with and
> one
> > without) -- are we simply checking if the tagged field exists in the
> > message before we try to read it?
> > I think this is the sort of thing we already do, but just wanted to
> > clarify. We don't need to distinguish versions since we just need to
> > distinguish if the tagged field is there or not.
> >
> > Thanks,
> > Justine
> >
> >
> > On Tue, Mar 28, 2023 at 3:42 PM Jeff Kim 
> > wrote:
> >
> > > Hi Alexandre,
> > >
> > > Thanks for the feedback.
> > >
> > > 100. You're correct. I updated the KIP to explicitly state that once we
> > > introduce a new tagged field in a later version, that version can never
> > be
> > > downgraded to the listed versions.
> > >
> > > 101. I don't think there is a way to differentiate the two but also not
> > > sure if there will be a need to.
> > >
> > > Best,
> > > Jeff
> > >
> > > On Tue, Mar 28, 2023 at 2:13 PM Alexandre Dupriez <
> > > alexandre.dupr...@gmail.com> wrote:
> > >
> > > > Hi Jeff,
> > > >
> > > > Thank you for the fast answer!
> > > >
> > > > 100. Got it, I think I am starting to understand based on your
> example
> > > > exposing three Kafka versions. Please let me know if the following
> > > > corresponds to the correct interpretation.
> > > >
> > > > Taking OffsetCommitValue as an example, the idea is to bump the
> schema
> > > > to version 4 to make the record flexible. The schema version 4 will
> be
> > > > back-ported to versions 3.0.3, 3.1.3, 3.2.4, 3.3.3, 3.4.1 and
> > > > potentially 3.5. These versions will continue to serialise the record
> > > > with version 3 so that downgrades from there to earlier minor and/or
> > > > major versions are possible. And, these versions will deserialize the
> > > > records with version 4 so that they can support flexible records,
> > > > although they do not make use of tagged fields (if any is present).
> > > >
> > > > Then, in a future version, 3.x, x >= 6, a tag field will be added to
> > > > the record - say, topic id. Downgrade from 3.x to 3.0.3, 3.1.3,
> 3.2.4,
> > > > 3.3.3, 3.4.1 and 3.5 will be possible because these versions will
> > > > perform deserialisation with the record schema version 4, that is,
> > > > supporting tagged fields.
> > > >
> > > > Hence, this approach is both backward and forward looking and allows
> > > > to extend the scope of compatible versions for downgrades.
> > > >
> > > > [N] 3.x, x >= 6 --> downgradable to [I] 3.0.3, 3.1.3, 3.2.4, 3.3.3,
> > > > 3.4.1, 3.5 --> downgradable to e.g. [O] 3.0.0.
> > > >
> > > > One note though is that a transitive downgrade from 3.x, x >= 6, to
> > > > 3.0.0 via the version domain [I], will not be supported. Should it be
> > > > explicitly mentioned in the KIP that downgrades from 3.0.3, 3.1.3,
> > > > 3.2.4, 3.3.3, 3.4.1, 3.5 to earlier versions may not be possible (if
> > > > offsets or group metadata records were generated by a higher version
> > > > 3.x, x >= 6)? Or am I still misunderstanding?
> > > >
> > > > 101. I will try to provide an example to illustrate what I mean by
> > > > version. Consider the GroupMetadataValue schema version 4 and the
> > > > tagged field topicId introduced in 3.6. Let's say a new optional
> field
> > > > needs to be added in 3.7. We will have two records version 4, one
> with
> > > > topic id, the other with topic id and the new field. The new field is
> > > > semantically optional (structurally it is always optional since it is
> > > > a tagged field) but we want to make the distinction between a record
> > > > generated by 3.6 and one generated by 3.7. How do we resolve the
> > > > ambiguity?
> > > >
> > > > Thanks!
> > > > Alexandre
> > > >
> > > > Le mar. 28 mars 2023 à 16:13, Jeff Kim  >
> > a
> > > > écrit :
> > > > >
> > > > > Hi Alexandre,
> > > > >
> > > > > Thank you for taking a look!
> > > > >
> > > > > 100. I am not sure I fully understand what you mean by forcefully
> > > adding
> > > > > tagged fields. Let's say VX does not have a flexible version,
> > > > > VY allows deserialization but serializes with a non-flexible
> version,
> > > and
> > > > > VZ introduces a new tagged field.
> > > > > VX upgrade to VY then downgrade back to VX works because even if
> > group
> > > > > metadata changes VY will 

Re: [DISCUSS] KIP-915: Next Gen Group Coordinator Downgrade Path

[Jeff Kim]

Hi Justine,

Thank you for the comment. Yes, that is what we do today. Let's say there's
a record
that has two tagged fields, x and y. If the coordinator's schema only
supports field x, that will be the only field it reads.

Thanks,
Jeff

On Wed, Mar 29, 2023 at 12:43 PM Justine Olshan
 wrote:

> Hey Jeff,
>
> Going off of Alexandre's point -- if we have a tagged field in one version
> but not another (or even in the same version we could have one with and one
> without) -- are we simply checking if the tagged field exists in the
> message before we try to read it?
> I think this is the sort of thing we already do, but just wanted to
> clarify. We don't need to distinguish versions since we just need to
> distinguish if the tagged field is there or not.
>
> Thanks,
> Justine
>
>
> On Tue, Mar 28, 2023 at 3:42 PM Jeff Kim 
> wrote:
>
> > Hi Alexandre,
> >
> > Thanks for the feedback.
> >
> > 100. You're correct. I updated the KIP to explicitly state that once we
> > introduce a new tagged field in a later version, that version can never
> be
> > downgraded to the listed versions.
> >
> > 101. I don't think there is a way to differentiate the two but also not
> > sure if there will be a need to.
> >
> > Best,
> > Jeff
> >
> > On Tue, Mar 28, 2023 at 2:13 PM Alexandre Dupriez <
> > alexandre.dupr...@gmail.com> wrote:
> >
> > > Hi Jeff,
> > >
> > > Thank you for the fast answer!
> > >
> > > 100. Got it, I think I am starting to understand based on your example
> > > exposing three Kafka versions. Please let me know if the following
> > > corresponds to the correct interpretation.
> > >
> > > Taking OffsetCommitValue as an example, the idea is to bump the schema
> > > to version 4 to make the record flexible. The schema version 4 will be
> > > back-ported to versions 3.0.3, 3.1.3, 3.2.4, 3.3.3, 3.4.1 and
> > > potentially 3.5. These versions will continue to serialise the record
> > > with version 3 so that downgrades from there to earlier minor and/or
> > > major versions are possible. And, these versions will deserialize the
> > > records with version 4 so that they can support flexible records,
> > > although they do not make use of tagged fields (if any is present).
> > >
> > > Then, in a future version, 3.x, x >= 6, a tag field will be added to
> > > the record - say, topic id. Downgrade from 3.x to 3.0.3, 3.1.3, 3.2.4,
> > > 3.3.3, 3.4.1 and 3.5 will be possible because these versions will
> > > perform deserialisation with the record schema version 4, that is,
> > > supporting tagged fields.
> > >
> > > Hence, this approach is both backward and forward looking and allows
> > > to extend the scope of compatible versions for downgrades.
> > >
> > > [N] 3.x, x >= 6 --> downgradable to [I] 3.0.3, 3.1.3, 3.2.4, 3.3.3,
> > > 3.4.1, 3.5 --> downgradable to e.g. [O] 3.0.0.
> > >
> > > One note though is that a transitive downgrade from 3.x, x >= 6, to
> > > 3.0.0 via the version domain [I], will not be supported. Should it be
> > > explicitly mentioned in the KIP that downgrades from 3.0.3, 3.1.3,
> > > 3.2.4, 3.3.3, 3.4.1, 3.5 to earlier versions may not be possible (if
> > > offsets or group metadata records were generated by a higher version
> > > 3.x, x >= 6)? Or am I still misunderstanding?
> > >
> > > 101. I will try to provide an example to illustrate what I mean by
> > > version. Consider the GroupMetadataValue schema version 4 and the
> > > tagged field topicId introduced in 3.6. Let's say a new optional field
> > > needs to be added in 3.7. We will have two records version 4, one with
> > > topic id, the other with topic id and the new field. The new field is
> > > semantically optional (structurally it is always optional since it is
> > > a tagged field) but we want to make the distinction between a record
> > > generated by 3.6 and one generated by 3.7. How do we resolve the
> > > ambiguity?
> > >
> > > Thanks!
> > > Alexandre
> > >
> > > Le mar. 28 mars 2023 à 16:13, Jeff Kim 
> a
> > > écrit :
> > > >
> > > > Hi Alexandre,
> > > >
> > > > Thank you for taking a look!
> > > >
> > > > 100. I am not sure I fully understand what you mean by forcefully
> > adding
> > > > tagged fields. Let's say VX does not have a flexible version,
> > > > VY allows deserialization but serializes with a non-flexible version,
> > and
> > > > VZ introduces a new tagged field.
> > > > VX upgrade to VY then downgrade back to VX works because even if
> group
> > > > metadata changes VY will serialize with
> > > > the highest non-flexible version. VZ to VY back to VZ also works
> > because
> > > > even if VY serializes with a non-flexible field
> > > > VZ will be able to deserialize it as it is a supported version. Does
> > this
> > > > answer your question?
> > > >
> > > > 101. The future versioning scheme needs to be backward compatible
> with
> > > > older coordinators. Wouldn't segregating into 2 versions
> > > > be incompatible?
> > > >
> > > > Thanks,
> > > > Jeff
> > > >
> > > > On Tue, Mar 28, 2023 at 5:47 AM 

Re: [DISCUSS] KIP-915: Next Gen Group Coordinator Downgrade Path

[Justine Olshan]

Hey Jeff,

Going off of Alexandre's point -- if we have a tagged field in one version
but not another (or even in the same version we could have one with and one
without) -- are we simply checking if the tagged field exists in the
message before we try to read it?
I think this is the sort of thing we already do, but just wanted to
clarify. We don't need to distinguish versions since we just need to
distinguish if the tagged field is there or not.

Thanks,
Justine


On Tue, Mar 28, 2023 at 3:42 PM Jeff Kim 
wrote:

> Hi Alexandre,
>
> Thanks for the feedback.
>
> 100. You're correct. I updated the KIP to explicitly state that once we
> introduce a new tagged field in a later version, that version can never be
> downgraded to the listed versions.
>
> 101. I don't think there is a way to differentiate the two but also not
> sure if there will be a need to.
>
> Best,
> Jeff
>
> On Tue, Mar 28, 2023 at 2:13 PM Alexandre Dupriez <
> alexandre.dupr...@gmail.com> wrote:
>
> > Hi Jeff,
> >
> > Thank you for the fast answer!
> >
> > 100. Got it, I think I am starting to understand based on your example
> > exposing three Kafka versions. Please let me know if the following
> > corresponds to the correct interpretation.
> >
> > Taking OffsetCommitValue as an example, the idea is to bump the schema
> > to version 4 to make the record flexible. The schema version 4 will be
> > back-ported to versions 3.0.3, 3.1.3, 3.2.4, 3.3.3, 3.4.1 and
> > potentially 3.5. These versions will continue to serialise the record
> > with version 3 so that downgrades from there to earlier minor and/or
> > major versions are possible. And, these versions will deserialize the
> > records with version 4 so that they can support flexible records,
> > although they do not make use of tagged fields (if any is present).
> >
> > Then, in a future version, 3.x, x >= 6, a tag field will be added to
> > the record - say, topic id. Downgrade from 3.x to 3.0.3, 3.1.3, 3.2.4,
> > 3.3.3, 3.4.1 and 3.5 will be possible because these versions will
> > perform deserialisation with the record schema version 4, that is,
> > supporting tagged fields.
> >
> > Hence, this approach is both backward and forward looking and allows
> > to extend the scope of compatible versions for downgrades.
> >
> > [N] 3.x, x >= 6 --> downgradable to [I] 3.0.3, 3.1.3, 3.2.4, 3.3.3,
> > 3.4.1, 3.5 --> downgradable to e.g. [O] 3.0.0.
> >
> > One note though is that a transitive downgrade from 3.x, x >= 6, to
> > 3.0.0 via the version domain [I], will not be supported. Should it be
> > explicitly mentioned in the KIP that downgrades from 3.0.3, 3.1.3,
> > 3.2.4, 3.3.3, 3.4.1, 3.5 to earlier versions may not be possible (if
> > offsets or group metadata records were generated by a higher version
> > 3.x, x >= 6)? Or am I still misunderstanding?
> >
> > 101. I will try to provide an example to illustrate what I mean by
> > version. Consider the GroupMetadataValue schema version 4 and the
> > tagged field topicId introduced in 3.6. Let's say a new optional field
> > needs to be added in 3.7. We will have two records version 4, one with
> > topic id, the other with topic id and the new field. The new field is
> > semantically optional (structurally it is always optional since it is
> > a tagged field) but we want to make the distinction between a record
> > generated by 3.6 and one generated by 3.7. How do we resolve the
> > ambiguity?
> >
> > Thanks!
> > Alexandre
> >
> > Le mar. 28 mars 2023 à 16:13, Jeff Kim  a
> > écrit :
> > >
> > > Hi Alexandre,
> > >
> > > Thank you for taking a look!
> > >
> > > 100. I am not sure I fully understand what you mean by forcefully
> adding
> > > tagged fields. Let's say VX does not have a flexible version,
> > > VY allows deserialization but serializes with a non-flexible version,
> and
> > > VZ introduces a new tagged field.
> > > VX upgrade to VY then downgrade back to VX works because even if group
> > > metadata changes VY will serialize with
> > > the highest non-flexible version. VZ to VY back to VZ also works
> because
> > > even if VY serializes with a non-flexible field
> > > VZ will be able to deserialize it as it is a supported version. Does
> this
> > > answer your question?
> > >
> > > 101. The future versioning scheme needs to be backward compatible with
> > > older coordinators. Wouldn't segregating into 2 versions
> > > be incompatible?
> > >
> > > Thanks,
> > > Jeff
> > >
> > > On Tue, Mar 28, 2023 at 5:47 AM Alexandre Dupriez <
> > > alexandre.dupr...@gmail.com> wrote:
> > >
> > > > Hi Jeff, Team,
> > > >
> > > > Thank you for the KIP. This is a very interesting approach. I feel it
> > > > is simpler than the described alternative although it comes with
> > > > tradeoffs, thanks for highlighting those. If I may, I would like to
> > > > share two naive questions.
> > > >
> > > > 100. The KIP mentions that records will be serialised with the
> highest
> > > > non-flexible version (e.g. 3 for GroupMetadataValue and
> > > > 

Re: [DISCUSS] KIP-915: Next Gen Group Coordinator Downgrade Path

[Jeff Kim]

Hi Alexandre,

Thanks for the feedback.

100. You're correct. I updated the KIP to explicitly state that once we
introduce a new tagged field in a later version, that version can never be
downgraded to the listed versions.

101. I don't think there is a way to differentiate the two but also not
sure if there will be a need to.

Best,
Jeff

On Tue, Mar 28, 2023 at 2:13 PM Alexandre Dupriez <
alexandre.dupr...@gmail.com> wrote:

> Hi Jeff,
>
> Thank you for the fast answer!
>
> 100. Got it, I think I am starting to understand based on your example
> exposing three Kafka versions. Please let me know if the following
> corresponds to the correct interpretation.
>
> Taking OffsetCommitValue as an example, the idea is to bump the schema
> to version 4 to make the record flexible. The schema version 4 will be
> back-ported to versions 3.0.3, 3.1.3, 3.2.4, 3.3.3, 3.4.1 and
> potentially 3.5. These versions will continue to serialise the record
> with version 3 so that downgrades from there to earlier minor and/or
> major versions are possible. And, these versions will deserialize the
> records with version 4 so that they can support flexible records,
> although they do not make use of tagged fields (if any is present).
>
> Then, in a future version, 3.x, x >= 6, a tag field will be added to
> the record - say, topic id. Downgrade from 3.x to 3.0.3, 3.1.3, 3.2.4,
> 3.3.3, 3.4.1 and 3.5 will be possible because these versions will
> perform deserialisation with the record schema version 4, that is,
> supporting tagged fields.
>
> Hence, this approach is both backward and forward looking and allows
> to extend the scope of compatible versions for downgrades.
>
> [N] 3.x, x >= 6 --> downgradable to [I] 3.0.3, 3.1.3, 3.2.4, 3.3.3,
> 3.4.1, 3.5 --> downgradable to e.g. [O] 3.0.0.
>
> One note though is that a transitive downgrade from 3.x, x >= 6, to
> 3.0.0 via the version domain [I], will not be supported. Should it be
> explicitly mentioned in the KIP that downgrades from 3.0.3, 3.1.3,
> 3.2.4, 3.3.3, 3.4.1, 3.5 to earlier versions may not be possible (if
> offsets or group metadata records were generated by a higher version
> 3.x, x >= 6)? Or am I still misunderstanding?
>
> 101. I will try to provide an example to illustrate what I mean by
> version. Consider the GroupMetadataValue schema version 4 and the
> tagged field topicId introduced in 3.6. Let's say a new optional field
> needs to be added in 3.7. We will have two records version 4, one with
> topic id, the other with topic id and the new field. The new field is
> semantically optional (structurally it is always optional since it is
> a tagged field) but we want to make the distinction between a record
> generated by 3.6 and one generated by 3.7. How do we resolve the
> ambiguity?
>
> Thanks!
> Alexandre
>
> Le mar. 28 mars 2023 à 16:13, Jeff Kim  a
> écrit :
> >
> > Hi Alexandre,
> >
> > Thank you for taking a look!
> >
> > 100. I am not sure I fully understand what you mean by forcefully adding
> > tagged fields. Let's say VX does not have a flexible version,
> > VY allows deserialization but serializes with a non-flexible version, and
> > VZ introduces a new tagged field.
> > VX upgrade to VY then downgrade back to VX works because even if group
> > metadata changes VY will serialize with
> > the highest non-flexible version. VZ to VY back to VZ also works because
> > even if VY serializes with a non-flexible field
> > VZ will be able to deserialize it as it is a supported version. Does this
> > answer your question?
> >
> > 101. The future versioning scheme needs to be backward compatible with
> > older coordinators. Wouldn't segregating into 2 versions
> > be incompatible?
> >
> > Thanks,
> > Jeff
> >
> > On Tue, Mar 28, 2023 at 5:47 AM Alexandre Dupriez <
> > alexandre.dupr...@gmail.com> wrote:
> >
> > > Hi Jeff, Team,
> > >
> > > Thank you for the KIP. This is a very interesting approach. I feel it
> > > is simpler than the described alternative although it comes with
> > > tradeoffs, thanks for highlighting those. If I may, I would like to
> > > share two naive questions.
> > >
> > > 100. The KIP mentions that records will be serialised with the highest
> > > non-flexible version (e.g. 3 for GroupMetadataValue and
> > > OffsetCommitValue) so that records can be deserialized with earlier
> > > versions of Kafka. I am not sure I understand correctly: is the idea
> > > to forcefully add tagged fields at the end of the records while
> > > maintaining the existing version (3 for the two record types just
> > > mentioned) so that they can be deserialized by existing Kafka versions
> > > for which the version of these record types is not known as flexible,
> > > while at the same time preserving the new tagged fields to new Kafka
> > > versions abreast of the addition of a new flexible version for these
> > > record types? If so, is it "bypassing" the protocol convention which
> > > prescribes the use of a flexible version to allow the use of tagged
> > > 

Re: [DISCUSS] KIP-915: Next Gen Group Coordinator Downgrade Path

[Alexandre Dupriez]

Hi Jeff,

Thank you for the fast answer!

100. Got it, I think I am starting to understand based on your example
exposing three Kafka versions. Please let me know if the following
corresponds to the correct interpretation.

Taking OffsetCommitValue as an example, the idea is to bump the schema
to version 4 to make the record flexible. The schema version 4 will be
back-ported to versions 3.0.3, 3.1.3, 3.2.4, 3.3.3, 3.4.1 and
potentially 3.5. These versions will continue to serialise the record
with version 3 so that downgrades from there to earlier minor and/or
major versions are possible. And, these versions will deserialize the
records with version 4 so that they can support flexible records,
although they do not make use of tagged fields (if any is present).

Then, in a future version, 3.x, x >= 6, a tag field will be added to
the record - say, topic id. Downgrade from 3.x to 3.0.3, 3.1.3, 3.2.4,
3.3.3, 3.4.1 and 3.5 will be possible because these versions will
perform deserialisation with the record schema version 4, that is,
supporting tagged fields.

Hence, this approach is both backward and forward looking and allows
to extend the scope of compatible versions for downgrades.

[N] 3.x, x >= 6 --> downgradable to [I] 3.0.3, 3.1.3, 3.2.4, 3.3.3,
3.4.1, 3.5 --> downgradable to e.g. [O] 3.0.0.

One note though is that a transitive downgrade from 3.x, x >= 6, to
3.0.0 via the version domain [I], will not be supported. Should it be
explicitly mentioned in the KIP that downgrades from 3.0.3, 3.1.3,
3.2.4, 3.3.3, 3.4.1, 3.5 to earlier versions may not be possible (if
offsets or group metadata records were generated by a higher version
3.x, x >= 6)? Or am I still misunderstanding?

101. I will try to provide an example to illustrate what I mean by
version. Consider the GroupMetadataValue schema version 4 and the
tagged field topicId introduced in 3.6. Let's say a new optional field
needs to be added in 3.7. We will have two records version 4, one with
topic id, the other with topic id and the new field. The new field is
semantically optional (structurally it is always optional since it is
a tagged field) but we want to make the distinction between a record
generated by 3.6 and one generated by 3.7. How do we resolve the
ambiguity?

Thanks!
Alexandre

Le mar. 28 mars 2023 à 16:13, Jeff Kim  a écrit :
>
> Hi Alexandre,
>
> Thank you for taking a look!
>
> 100. I am not sure I fully understand what you mean by forcefully adding
> tagged fields. Let's say VX does not have a flexible version,
> VY allows deserialization but serializes with a non-flexible version, and
> VZ introduces a new tagged field.
> VX upgrade to VY then downgrade back to VX works because even if group
> metadata changes VY will serialize with
> the highest non-flexible version. VZ to VY back to VZ also works because
> even if VY serializes with a non-flexible field
> VZ will be able to deserialize it as it is a supported version. Does this
> answer your question?
>
> 101. The future versioning scheme needs to be backward compatible with
> older coordinators. Wouldn't segregating into 2 versions
> be incompatible?
>
> Thanks,
> Jeff
>
> On Tue, Mar 28, 2023 at 5:47 AM Alexandre Dupriez <
> alexandre.dupr...@gmail.com> wrote:
>
> > Hi Jeff, Team,
> >
> > Thank you for the KIP. This is a very interesting approach. I feel it
> > is simpler than the described alternative although it comes with
> > tradeoffs, thanks for highlighting those. If I may, I would like to
> > share two naive questions.
> >
> > 100. The KIP mentions that records will be serialised with the highest
> > non-flexible version (e.g. 3 for GroupMetadataValue and
> > OffsetCommitValue) so that records can be deserialized with earlier
> > versions of Kafka. I am not sure I understand correctly: is the idea
> > to forcefully add tagged fields at the end of the records while
> > maintaining the existing version (3 for the two record types just
> > mentioned) so that they can be deserialized by existing Kafka versions
> > for which the version of these record types is not known as flexible,
> > while at the same time preserving the new tagged fields to new Kafka
> > versions abreast of the addition of a new flexible version for these
> > record types? If so, is it "bypassing" the protocol convention which
> > prescribes the use of a flexible version to allow the use of tagged
> > fields?
> >
> > 101. After the bump of the records to a new version indicated as
> > flexible, the record version is expected to never change while the
> > underlying tagged fields could potentially still evolve over time. One
> > potential downside is that we lose the benefits of the versioning
> > scheme enforced by the serde protocol. Could this become a problem in
> > the future if there is ever a need to segregate two distinct
> > "versions" of the appended record structure held by the tagged fields?
> >
> > Thanks,
> > Alexandre
> >
> > Le jeu. 23 mars 2023 à 18:15, Jeff Kim  a
> > écrit :
> > >
> > > 

Re: [DISCUSS] KIP-915: Next Gen Group Coordinator Downgrade Path

[Jeff Kim]

Hi Alexandre,

Thank you for taking a look!

100. I am not sure I fully understand what you mean by forcefully adding
tagged fields. Let's say VX does not have a flexible version,
VY allows deserialization but serializes with a non-flexible version, and
VZ introduces a new tagged field.
VX upgrade to VY then downgrade back to VX works because even if group
metadata changes VY will serialize with
the highest non-flexible version. VZ to VY back to VZ also works because
even if VY serializes with a non-flexible field
VZ will be able to deserialize it as it is a supported version. Does this
answer your question?

101. The future versioning scheme needs to be backward compatible with
older coordinators. Wouldn't segregating into 2 versions
be incompatible?

Thanks,
Jeff

On Tue, Mar 28, 2023 at 5:47 AM Alexandre Dupriez <
alexandre.dupr...@gmail.com> wrote:

> Hi Jeff, Team,
>
> Thank you for the KIP. This is a very interesting approach. I feel it
> is simpler than the described alternative although it comes with
> tradeoffs, thanks for highlighting those. If I may, I would like to
> share two naive questions.
>
> 100. The KIP mentions that records will be serialised with the highest
> non-flexible version (e.g. 3 for GroupMetadataValue and
> OffsetCommitValue) so that records can be deserialized with earlier
> versions of Kafka. I am not sure I understand correctly: is the idea
> to forcefully add tagged fields at the end of the records while
> maintaining the existing version (3 for the two record types just
> mentioned) so that they can be deserialized by existing Kafka versions
> for which the version of these record types is not known as flexible,
> while at the same time preserving the new tagged fields to new Kafka
> versions abreast of the addition of a new flexible version for these
> record types? If so, is it "bypassing" the protocol convention which
> prescribes the use of a flexible version to allow the use of tagged
> fields?
>
> 101. After the bump of the records to a new version indicated as
> flexible, the record version is expected to never change while the
> underlying tagged fields could potentially still evolve over time. One
> potential downside is that we lose the benefits of the versioning
> scheme enforced by the serde protocol. Could this become a problem in
> the future if there is ever a need to segregate two distinct
> "versions" of the appended record structure held by the tagged fields?
>
> Thanks,
> Alexandre
>
> Le jeu. 23 mars 2023 à 18:15, Jeff Kim  a
> écrit :
> >
> > Hi Yi,
> >
> > > Does it mean with a flexible version, the future
> > version of these value types will stay at the version where the
> flexibility
> > is first introduced? Will there be any need to bump the version again in
> > the future?
> >
> > Yes, there will be no need to bump the version since we will only be
> adding
> > tagged fields but in the chance the version is bumped, we will
> deserialize
> > to the highest known (flexible) version which will ignore unknown tagged
> > fields.
> >
> > > To enforce the version not bumping, is it possible to have a unit test
> to
> > gate?
> >
> > Do you have some tests in mind? I don't think we can tell whether a
> version
> > was bumped or not.
> >
> > Best,
> > Jeff
> >
> > On Thu, Mar 23, 2023 at 12:07 PM Yi Ding 
> wrote:
> >
> > > Hi Jeff,
> > >
> > > Thanks for the update. Does it mean with a flexible version, the future
> > > version of these value types will stay at the version where the
> flexibility
> > > is first introduced? Will there be any need to bump the version again
> in
> > > the future?
> > > To enforce the version not bumping, is it possible to have a unit test
> to
> > > gate?
> > >
> > >
> > > On Wed, Mar 22, 2023 at 3:19 PM Jeff Kim  >
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > After discussing with my colleagues, I have repurposed the KIP for a
> > > > general downgrade solution for both transaction and group
> coordinators.
> > > The
> > > > KIP no longer discusses the downgrade path but instead lays out the
> > > > foundation for future downgrade solutions.
> > > >
> > > > Link:
> > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-915%3A+Txn+and+Group+Coordinator+Downgrade+Foundation
> > > >
> > > > Thanks,
> > > > Jeff
> > > >
> > > > On Mon, Mar 20, 2023 at 7:37 PM Jeff Kim 
> wrote:
> > > >
> > > > > Hi David and Justine,
> > > > >
> > > > > Thank you both for the detailed feedback.
> > > > >
> > > > > David,
> > > > >
> > > > > 1. That makes sense. I revised the "Reading new fields" section
> with
> > > how
> > > > > we can downgrade to the highest known version and that this was
> > > confirmed
> > > > > via unit testing. I also attempted to dive deeper into using tagged
> > > > fields
> > > > > and the rejected alternative. Please let me know what you think.
> > > > >
> > > > > 2. Under "Restrictions and Guidelines" I updated the paragraph to
> > > clearly
> > > > > state that we want to use tagged 

Re: [DISCUSS] KIP-915: Next Gen Group Coordinator Downgrade Path

[Alexandre Dupriez]

Hi Jeff, Team,

Thank you for the KIP. This is a very interesting approach. I feel it
is simpler than the described alternative although it comes with
tradeoffs, thanks for highlighting those. If I may, I would like to
share two naive questions.

100. The KIP mentions that records will be serialised with the highest
non-flexible version (e.g. 3 for GroupMetadataValue and
OffsetCommitValue) so that records can be deserialized with earlier
versions of Kafka. I am not sure I understand correctly: is the idea
to forcefully add tagged fields at the end of the records while
maintaining the existing version (3 for the two record types just
mentioned) so that they can be deserialized by existing Kafka versions
for which the version of these record types is not known as flexible,
while at the same time preserving the new tagged fields to new Kafka
versions abreast of the addition of a new flexible version for these
record types? If so, is it "bypassing" the protocol convention which
prescribes the use of a flexible version to allow the use of tagged
fields?

101. After the bump of the records to a new version indicated as
flexible, the record version is expected to never change while the
underlying tagged fields could potentially still evolve over time. One
potential downside is that we lose the benefits of the versioning
scheme enforced by the serde protocol. Could this become a problem in
the future if there is ever a need to segregate two distinct
"versions" of the appended record structure held by the tagged fields?

Thanks,
Alexandre

Le jeu. 23 mars 2023 à 18:15, Jeff Kim  a écrit :
>
> Hi Yi,
>
> > Does it mean with a flexible version, the future
> version of these value types will stay at the version where the flexibility
> is first introduced? Will there be any need to bump the version again in
> the future?
>
> Yes, there will be no need to bump the version since we will only be adding
> tagged fields but in the chance the version is bumped, we will deserialize
> to the highest known (flexible) version which will ignore unknown tagged
> fields.
>
> > To enforce the version not bumping, is it possible to have a unit test to
> gate?
>
> Do you have some tests in mind? I don't think we can tell whether a version
> was bumped or not.
>
> Best,
> Jeff
>
> On Thu, Mar 23, 2023 at 12:07 PM Yi Ding  wrote:
>
> > Hi Jeff,
> >
> > Thanks for the update. Does it mean with a flexible version, the future
> > version of these value types will stay at the version where the flexibility
> > is first introduced? Will there be any need to bump the version again in
> > the future?
> > To enforce the version not bumping, is it possible to have a unit test to
> > gate?
> >
> >
> > On Wed, Mar 22, 2023 at 3:19 PM Jeff Kim 
> > wrote:
> >
> > > Hi all,
> > >
> > > After discussing with my colleagues, I have repurposed the KIP for a
> > > general downgrade solution for both transaction and group coordinators.
> > The
> > > KIP no longer discusses the downgrade path but instead lays out the
> > > foundation for future downgrade solutions.
> > >
> > > Link:
> > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-915%3A+Txn+and+Group+Coordinator+Downgrade+Foundation
> > >
> > > Thanks,
> > > Jeff
> > >
> > > On Mon, Mar 20, 2023 at 7:37 PM Jeff Kim  wrote:
> > >
> > > > Hi David and Justine,
> > > >
> > > > Thank you both for the detailed feedback.
> > > >
> > > > David,
> > > >
> > > > 1. That makes sense. I revised the "Reading new fields" section with
> > how
> > > > we can downgrade to the highest known version and that this was
> > confirmed
> > > > via unit testing. I also attempted to dive deeper into using tagged
> > > fields
> > > > and the rejected alternative. Please let me know what you think.
> > > >
> > > > 2. Under "Restrictions and Guidelines" I updated the paragraph to
> > clearly
> > > > state that we want to use tagged fields across all record types
> > > introduced
> > > > in KIP-848 including OffsetCommitValue.
> > > >
> > > > 3. Would it be possible to bump the OffsetCommitValue record version to
> > > > make it flexible along with the changes to parse with the highest known
> > > > version? I'm not sure I understand why we cannot make both changes
> > > together.
> > > >
> > > > 4. I completely missed this. Added some notes at the end of
> > "Restrictions
> > > > and Guidelines". Unfortunately I can't think of a solution at the
> > moment.
> > > > Will get back to you.
> > > >
> > > > 5. I have a section under "Identifying New Record Types" that discusses
> > > > this:
> > > >  > We can automate the cleanup by writing tombstones when the
> > coordinator
> > > > reads unrecognizable records. This may add duplicate work if tombstones
> > > > were already added but not yet pruned by the log cleaner.
> > > > This is a sure way to delete any unknown record types even if the
> > > operator
> > > > does not follow the steps.
> > > >
> > > > 6. Thanks, I have expanded on the section on transactional 

Re: [DISCUSS] KIP-915: Next Gen Group Coordinator Downgrade Path

[Jeff Kim]

Hi Yi,

> Does it mean with a flexible version, the future
version of these value types will stay at the version where the flexibility
is first introduced? Will there be any need to bump the version again in
the future?

Yes, there will be no need to bump the version since we will only be adding
tagged fields but in the chance the version is bumped, we will deserialize
to the highest known (flexible) version which will ignore unknown tagged
fields.

> To enforce the version not bumping, is it possible to have a unit test to
gate?

Do you have some tests in mind? I don't think we can tell whether a version
was bumped or not.

Best,
Jeff

On Thu, Mar 23, 2023 at 12:07 PM Yi Ding  wrote:

> Hi Jeff,
>
> Thanks for the update. Does it mean with a flexible version, the future
> version of these value types will stay at the version where the flexibility
> is first introduced? Will there be any need to bump the version again in
> the future?
> To enforce the version not bumping, is it possible to have a unit test to
> gate?
>
>
> On Wed, Mar 22, 2023 at 3:19 PM Jeff Kim 
> wrote:
>
> > Hi all,
> >
> > After discussing with my colleagues, I have repurposed the KIP for a
> > general downgrade solution for both transaction and group coordinators.
> The
> > KIP no longer discusses the downgrade path but instead lays out the
> > foundation for future downgrade solutions.
> >
> > Link:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-915%3A+Txn+and+Group+Coordinator+Downgrade+Foundation
> >
> > Thanks,
> > Jeff
> >
> > On Mon, Mar 20, 2023 at 7:37 PM Jeff Kim  wrote:
> >
> > > Hi David and Justine,
> > >
> > > Thank you both for the detailed feedback.
> > >
> > > David,
> > >
> > > 1. That makes sense. I revised the "Reading new fields" section with
> how
> > > we can downgrade to the highest known version and that this was
> confirmed
> > > via unit testing. I also attempted to dive deeper into using tagged
> > fields
> > > and the rejected alternative. Please let me know what you think.
> > >
> > > 2. Under "Restrictions and Guidelines" I updated the paragraph to
> clearly
> > > state that we want to use tagged fields across all record types
> > introduced
> > > in KIP-848 including OffsetCommitValue.
> > >
> > > 3. Would it be possible to bump the OffsetCommitValue record version to
> > > make it flexible along with the changes to parse with the highest known
> > > version? I'm not sure I understand why we cannot make both changes
> > together.
> > >
> > > 4. I completely missed this. Added some notes at the end of
> "Restrictions
> > > and Guidelines". Unfortunately I can't think of a solution at the
> moment.
> > > Will get back to you.
> > >
> > > 5. I have a section under "Identifying New Record Types" that discusses
> > > this:
> > >  > We can automate the cleanup by writing tombstones when the
> coordinator
> > > reads unrecognizable records. This may add duplicate work if tombstones
> > > were already added but not yet pruned by the log cleaner.
> > > This is a sure way to delete any unknown record types even if the
> > operator
> > > does not follow the steps.
> > >
> > > 6. Thanks, I have expanded on the section on transactional offset
> > commits.
> > > As for log compaction, my understanding was that we can control the
> > process
> > > by forcing compaction. Is my understanding incorrect?
> > >
> > > 7. We throw an exception if ConfigResource.type is unknown which is
> true
> > > in our case because KIP-848 introduces a new GROUP resource type. We
> will
> > > need to add some sort of safeguard if the dynamic configs are not
> deleted
> > > before the server downgrade. I'll give you an update once I update the
> > > section.
> > >
> > > Justine,
> > >
> > > On the transactional offset commits, I have updated the KIP to reflect
> > > your points after our offline discussion (much appreciated). It seems
> > that
> > > the work required to abort from the server side is fairly big and will
> > > require additional investigation if we are to go down this path.
> > >
> > > As for downgrade limitations, I missed that part so thanks for the
> catch
> > > (along with David). Unfortunately, the proposed design won't allow
> > > downgrades even after the new record types are deleted because the
> > existing
> > > OffsetCommitValue record is not flexible and will not be able to parse
> > > the topicId tagged field. I'll think more about this and get back to
> you.
> > >
> > > Best,
> > > Jeff
> > >
> > > On Mon, Mar 20, 2023 at 2:16 PM Justine Olshan
> > >  wrote:
> > >
> > >> Hey Jeff and David,
> > >>
> > >> Thanks for the KIP! I was also looking into this a bit since I may
> want
> > to
> > >> change the record format for KIP-890 as well (finally implementing the
> > >> record change from KIP-360 to better support the epoch bump. This will
> > >> potentially be helpful for me to implement that work.
> > >>
> > >> I discussed some aspects with David offline. Namely, the alternative
> > >> solution 

Re: [DISCUSS] KIP-915: Next Gen Group Coordinator Downgrade Path

[Yi Ding]

Hi Jeff,

Thanks for the update. Does it mean with a flexible version, the future
version of these value types will stay at the version where the flexibility
is first introduced? Will there be any need to bump the version again in
the future?
To enforce the version not bumping, is it possible to have a unit test to
gate?


On Wed, Mar 22, 2023 at 3:19 PM Jeff Kim 
wrote:

> Hi all,
>
> After discussing with my colleagues, I have repurposed the KIP for a
> general downgrade solution for both transaction and group coordinators. The
> KIP no longer discusses the downgrade path but instead lays out the
> foundation for future downgrade solutions.
>
> Link:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-915%3A+Txn+and+Group+Coordinator+Downgrade+Foundation
>
> Thanks,
> Jeff
>
> On Mon, Mar 20, 2023 at 7:37 PM Jeff Kim  wrote:
>
> > Hi David and Justine,
> >
> > Thank you both for the detailed feedback.
> >
> > David,
> >
> > 1. That makes sense. I revised the "Reading new fields" section with how
> > we can downgrade to the highest known version and that this was confirmed
> > via unit testing. I also attempted to dive deeper into using tagged
> fields
> > and the rejected alternative. Please let me know what you think.
> >
> > 2. Under "Restrictions and Guidelines" I updated the paragraph to clearly
> > state that we want to use tagged fields across all record types
> introduced
> > in KIP-848 including OffsetCommitValue.
> >
> > 3. Would it be possible to bump the OffsetCommitValue record version to
> > make it flexible along with the changes to parse with the highest known
> > version? I'm not sure I understand why we cannot make both changes
> together.
> >
> > 4. I completely missed this. Added some notes at the end of "Restrictions
> > and Guidelines". Unfortunately I can't think of a solution at the moment.
> > Will get back to you.
> >
> > 5. I have a section under "Identifying New Record Types" that discusses
> > this:
> >  > We can automate the cleanup by writing tombstones when the coordinator
> > reads unrecognizable records. This may add duplicate work if tombstones
> > were already added but not yet pruned by the log cleaner.
> > This is a sure way to delete any unknown record types even if the
> operator
> > does not follow the steps.
> >
> > 6. Thanks, I have expanded on the section on transactional offset
> commits.
> > As for log compaction, my understanding was that we can control the
> process
> > by forcing compaction. Is my understanding incorrect?
> >
> > 7. We throw an exception if ConfigResource.type is unknown which is true
> > in our case because KIP-848 introduces a new GROUP resource type. We will
> > need to add some sort of safeguard if the dynamic configs are not deleted
> > before the server downgrade. I'll give you an update once I update the
> > section.
> >
> > Justine,
> >
> > On the transactional offset commits, I have updated the KIP to reflect
> > your points after our offline discussion (much appreciated). It seems
> that
> > the work required to abort from the server side is fairly big and will
> > require additional investigation if we are to go down this path.
> >
> > As for downgrade limitations, I missed that part so thanks for the catch
> > (along with David). Unfortunately, the proposed design won't allow
> > downgrades even after the new record types are deleted because the
> existing
> > OffsetCommitValue record is not flexible and will not be able to parse
> > the topicId tagged field. I'll think more about this and get back to you.
> >
> > Best,
> > Jeff
> >
> > On Mon, Mar 20, 2023 at 2:16 PM Justine Olshan
> >  wrote:
> >
> >> Hey Jeff and David,
> >>
> >> Thanks for the KIP! I was also looking into this a bit since I may want
> to
> >> change the record format for KIP-890 as well (finally implementing the
> >> record change from KIP-360 to better support the epoch bump. This will
> >> potentially be helpful for me to implement that work.
> >>
> >> I discussed some aspects with David offline. Namely, the alternative
> >> solution to rewrite the records in the old format upon downgrade. One
> plus
> >> to this approach is that it should trigger compaction for all the
> existing
> >> (new format) records.
> >> The main issue we ran into was how to handle ongoing transactions. (Ie,
> >> could we abort and rewrite with the old format) I think that's one of
> the
> >> main downsides. I think it could potentially be possible to do this, but
> >> we'd definitely need a server-side mechanism to abort and rewrite the
> >> records.
> >>
> >> I think the trouble I was running into with the current solution is that
> >> we
> >> can only downgrade to a version slightly before the new group
> coordinator.
> >> If someone was running 3.2 and they upgrade to 3.6, but find a bug in
> 3.4
> >> (or any version between 3.5 and the one they upgraded from), then they
> can
> >> only downgrade to 3.5. This puts us in a difficult spot. I guess in this
> >> scenario, 

Re: [DISCUSS] KIP-915: Next Gen Group Coordinator Downgrade Path

[Jeff Kim]

Hi all,

After discussing with my colleagues, I have repurposed the KIP for a
general downgrade solution for both transaction and group coordinators. The
KIP no longer discusses the downgrade path but instead lays out the
foundation for future downgrade solutions.

Link:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-915%3A+Txn+and+Group+Coordinator+Downgrade+Foundation

Thanks,
Jeff

On Mon, Mar 20, 2023 at 7:37 PM Jeff Kim  wrote:

> Hi David and Justine,
>
> Thank you both for the detailed feedback.
>
> David,
>
> 1. That makes sense. I revised the "Reading new fields" section with how
> we can downgrade to the highest known version and that this was confirmed
> via unit testing. I also attempted to dive deeper into using tagged fields
> and the rejected alternative. Please let me know what you think.
>
> 2. Under "Restrictions and Guidelines" I updated the paragraph to clearly
> state that we want to use tagged fields across all record types introduced
> in KIP-848 including OffsetCommitValue.
>
> 3. Would it be possible to bump the OffsetCommitValue record version to
> make it flexible along with the changes to parse with the highest known
> version? I'm not sure I understand why we cannot make both changes together.
>
> 4. I completely missed this. Added some notes at the end of "Restrictions
> and Guidelines". Unfortunately I can't think of a solution at the moment.
> Will get back to you.
>
> 5. I have a section under "Identifying New Record Types" that discusses
> this:
>  > We can automate the cleanup by writing tombstones when the coordinator
> reads unrecognizable records. This may add duplicate work if tombstones
> were already added but not yet pruned by the log cleaner.
> This is a sure way to delete any unknown record types even if the operator
> does not follow the steps.
>
> 6. Thanks, I have expanded on the section on transactional offset commits.
> As for log compaction, my understanding was that we can control the process
> by forcing compaction. Is my understanding incorrect?
>
> 7. We throw an exception if ConfigResource.type is unknown which is true
> in our case because KIP-848 introduces a new GROUP resource type. We will
> need to add some sort of safeguard if the dynamic configs are not deleted
> before the server downgrade. I'll give you an update once I update the
> section.
>
> Justine,
>
> On the transactional offset commits, I have updated the KIP to reflect
> your points after our offline discussion (much appreciated). It seems that
> the work required to abort from the server side is fairly big and will
> require additional investigation if we are to go down this path.
>
> As for downgrade limitations, I missed that part so thanks for the catch
> (along with David). Unfortunately, the proposed design won't allow
> downgrades even after the new record types are deleted because the existing
> OffsetCommitValue record is not flexible and will not be able to parse
> the topicId tagged field. I'll think more about this and get back to you.
>
> Best,
> Jeff
>
> On Mon, Mar 20, 2023 at 2:16 PM Justine Olshan
>  wrote:
>
>> Hey Jeff and David,
>>
>> Thanks for the KIP! I was also looking into this a bit since I may want to
>> change the record format for KIP-890 as well (finally implementing the
>> record change from KIP-360 to better support the epoch bump. This will
>> potentially be helpful for me to implement that work.
>>
>> I discussed some aspects with David offline. Namely, the alternative
>> solution to rewrite the records in the old format upon downgrade. One plus
>> to this approach is that it should trigger compaction for all the existing
>> (new format) records.
>> The main issue we ran into was how to handle ongoing transactions. (Ie,
>> could we abort and rewrite with the old format) I think that's one of the
>> main downsides. I think it could potentially be possible to do this, but
>> we'd definitely need a server-side mechanism to abort and rewrite the
>> records.
>>
>> I think the trouble I was running into with the current solution is that
>> we
>> can only downgrade to a version slightly before the new group coordinator.
>> If someone was running 3.2 and they upgrade to 3.6, but find a bug in 3.4
>> (or any version between 3.5 and the one they upgraded from), then they can
>> only downgrade to 3.5. This puts us in a difficult spot. I guess in this
>> scenario, they will need to wait for the new format records to get cleared
>> away before downgrading again. Is that correct?
>>
>> Thanks,
>> Justine
>>
>> On Thu, Mar 16, 2023 at 10:41 AM David Jacot > >
>> wrote:
>>
>> > Hi Jeff,
>> >
>> > Thanks for the KIP! I am really glad that we are finally addressing this
>> > gap in KIP-848. I have a few general comments.
>> >
>> > 1. Overall, I feel like the important bits are not bold enough in the
>> KIP.
>> > I think that it is good to explain the overall upgrade/downgrade process
>> > and to highlight where the issues are but I think that the core bits
>> 

Re: [DISCUSS] KIP-915: Next Gen Group Coordinator Downgrade Path

[Jeff Kim]

Hi David and Justine,

Thank you both for the detailed feedback.

David,

1. That makes sense. I revised the "Reading new fields" section with how we
can downgrade to the highest known version and that this was confirmed via
unit testing. I also attempted to dive deeper into using tagged fields and
the rejected alternative. Please let me know what you think.

2. Under "Restrictions and Guidelines" I updated the paragraph to clearly
state that we want to use tagged fields across all record types introduced
in KIP-848 including OffsetCommitValue.

3. Would it be possible to bump the OffsetCommitValue record version to
make it flexible along with the changes to parse with the highest known
version? I'm not sure I understand why we cannot make both changes together.

4. I completely missed this. Added some notes at the end of "Restrictions
and Guidelines". Unfortunately I can't think of a solution at the moment.
Will get back to you.

5. I have a section under "Identifying New Record Types" that discusses
this:
 > We can automate the cleanup by writing tombstones when the coordinator
reads unrecognizable records. This may add duplicate work if tombstones
were already added but not yet pruned by the log cleaner.
This is a sure way to delete any unknown record types even if the operator
does not follow the steps.

6. Thanks, I have expanded on the section on transactional offset commits.
As for log compaction, my understanding was that we can control the process
by forcing compaction. Is my understanding incorrect?

7. We throw an exception if ConfigResource.type is unknown which is true in
our case because KIP-848 introduces a new GROUP resource type. We will need
to add some sort of safeguard if the dynamic configs are not deleted before
the server downgrade. I'll give you an update once I update the section.

Justine,

On the transactional offset commits, I have updated the KIP to reflect your
points after our offline discussion (much appreciated). It seems that the
work required to abort from the server side is fairly big and will require
additional investigation if we are to go down this path.

As for downgrade limitations, I missed that part so thanks for the catch
(along with David). Unfortunately, the proposed design won't allow
downgrades even after the new record types are deleted because the existing
OffsetCommitValue record is not flexible and will not be able to parse
the topicId tagged field. I'll think more about this and get back to you.

Best,
Jeff

On Mon, Mar 20, 2023 at 2:16 PM Justine Olshan 
wrote:

> Hey Jeff and David,
>
> Thanks for the KIP! I was also looking into this a bit since I may want to
> change the record format for KIP-890 as well (finally implementing the
> record change from KIP-360 to better support the epoch bump. This will
> potentially be helpful for me to implement that work.
>
> I discussed some aspects with David offline. Namely, the alternative
> solution to rewrite the records in the old format upon downgrade. One plus
> to this approach is that it should trigger compaction for all the existing
> (new format) records.
> The main issue we ran into was how to handle ongoing transactions. (Ie,
> could we abort and rewrite with the old format) I think that's one of the
> main downsides. I think it could potentially be possible to do this, but
> we'd definitely need a server-side mechanism to abort and rewrite the
> records.
>
> I think the trouble I was running into with the current solution is that we
> can only downgrade to a version slightly before the new group coordinator.
> If someone was running 3.2 and they upgrade to 3.6, but find a bug in 3.4
> (or any version between 3.5 and the one they upgraded from), then they can
> only downgrade to 3.5. This puts us in a difficult spot. I guess in this
> scenario, they will need to wait for the new format records to get cleared
> away before downgrading again. Is that correct?
>
> Thanks,
> Justine
>
> On Thu, Mar 16, 2023 at 10:41 AM David Jacot 
> wrote:
>
> > Hi Jeff,
> >
> > Thanks for the KIP! I am really glad that we are finally addressing this
> > gap in KIP-848. I have a few general comments.
> >
> > 1. Overall, I feel like the important bits are not bold enough in the
> KIP.
> > I think that it is good to explain the overall upgrade/downgrade process
> > and to highlight where the issues are but I think that the core bits
> should
> > give more details. For instance, we should explain why relying on tag
> > fields works to ignore fields added in future releases. My understanding
> is
> > that it works because the buffer for the tagged fields is serialized at
> the
> > end so reading with the old version, which is a prefix of the new one,
> > works.
> >
> > 2. Moreover, it would be great if we could make the principle more
> general.
> > My hope is that we can keep reusing the principles introduced in the KIP
> in
> > future releases as well. For instance, let's say that we need to add a
> new
> > field to one 

Re: [DISCUSS] KIP-915: Next Gen Group Coordinator Downgrade Path

[Justine Olshan]

Hey Jeff and David,

Thanks for the KIP! I was also looking into this a bit since I may want to
change the record format for KIP-890 as well (finally implementing the
record change from KIP-360 to better support the epoch bump. This will
potentially be helpful for me to implement that work.

I discussed some aspects with David offline. Namely, the alternative
solution to rewrite the records in the old format upon downgrade. One plus
to this approach is that it should trigger compaction for all the existing
(new format) records.
The main issue we ran into was how to handle ongoing transactions. (Ie,
could we abort and rewrite with the old format) I think that's one of the
main downsides. I think it could potentially be possible to do this, but
we'd definitely need a server-side mechanism to abort and rewrite the
records.

I think the trouble I was running into with the current solution is that we
can only downgrade to a version slightly before the new group coordinator.
If someone was running 3.2 and they upgrade to 3.6, but find a bug in 3.4
(or any version between 3.5 and the one they upgraded from), then they can
only downgrade to 3.5. This puts us in a difficult spot. I guess in this
scenario, they will need to wait for the new format records to get cleared
away before downgrading again. Is that correct?

Thanks,
Justine

On Thu, Mar 16, 2023 at 10:41 AM David Jacot 
wrote:

> Hi Jeff,
>
> Thanks for the KIP! I am really glad that we are finally addressing this
> gap in KIP-848. I have a few general comments.
>
> 1. Overall, I feel like the important bits are not bold enough in the KIP.
> I think that it is good to explain the overall upgrade/downgrade process
> and to highlight where the issues are but I think that the core bits should
> give more details. For instance, we should explain why relying on tag
> fields works to ignore fields added in future releases. My understanding is
> that it works because the buffer for the tagged fields is serialized at the
> end so reading with the old version, which is a prefix of the new one,
> works.
>
> 2. Moreover, it would be great if we could make the principle more general.
> My hope is that we can keep reusing the principles introduced in the KIP in
> future releases as well. For instance, let's say that we need to add a new
> field to one of the new records introduced by KIP-848 or that we will have
> to introduce a new record type as well. Would it work for those cases as
> well?
>
> 3. Regarding enabling support for tagged fields for the OffsetCommitValue
> record, it would be great if you could give more details on the steps to
> get there in the KIP. My understanding is that we would have to do the
> following: 1) Update the code which reads the records to fail back to the
> highest known version if the version stored in the log is unknown. Let's
> say that we do this in AK 3.5. 2) We need to turn on tagged fields for the
> record. I think that we can only do this in AK 3.6+.
>
> 4. I may have missed this part but we should clearly explain the drawback
> of the proposed approach as well. Say that we enable tagged fields for
> OffsetCommitValue in AK 3.6. This means that it won't be possible to
> downgrade a cluster from 3.6 to a version earlier than 3.5. This is a
> significant limitation in my opinion because, I think, users don't
> necessarily upgrade to all versions.
>
> 5. In the proposal, it is not clear about whether the old software will
> delete unknown records or not. It is true that new records will be deleted
> when the group is downgraded but this only works if the operator respects
> the process.
>
> 6. It would be great if we could extend the rejected alternative. The
> alternative sounds clearly better when you read it so we should really
> explain the reason to reject it. 1) One issue that you mention is that the
> log must be compacted before downgrading and we don't really control this
> process. 2) Transactions may be difficult to handle. I suppose that it is
> possible to handle them though. Have you thought about this?
>
> 7. For the new dynamic configs, what happens if they are kept and the
> quorum controller is downgraded?
>
> Best,
> David
>
> On Thu, Mar 16, 2023 at 12:56 AM Jeff Kim 
> wrote:
>
> > Hi folks,
> >
> > I would like to start a discussion thread for KIP-915: Next Gen Group
> > Coordinator Downgrade Path which proposes the downgrade design for the
> new
> > group coordinator introduced in KIP-848
> > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-848%3A+The+Next+Generation+of+the+Consumer+Rebalance+Protocol
> > >
> > .
> >
> > KIP:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-915%3A+Next+Gen+Group+Coordinator+Downgrade+Path
> >
> > Thanks,
> > Jeff
> >
>

Re: [DISCUSS] KIP-915: Next Gen Group Coordinator Downgrade Path

[David Jacot]

Hi Jeff,

Thanks for the KIP! I am really glad that we are finally addressing this
gap in KIP-848. I have a few general comments.

1. Overall, I feel like the important bits are not bold enough in the KIP.
I think that it is good to explain the overall upgrade/downgrade process
and to highlight where the issues are but I think that the core bits should
give more details. For instance, we should explain why relying on tag
fields works to ignore fields added in future releases. My understanding is
that it works because the buffer for the tagged fields is serialized at the
end so reading with the old version, which is a prefix of the new one,
works.

2. Moreover, it would be great if we could make the principle more general.
My hope is that we can keep reusing the principles introduced in the KIP in
future releases as well. For instance, let's say that we need to add a new
field to one of the new records introduced by KIP-848 or that we will have
to introduce a new record type as well. Would it work for those cases as
well?

3. Regarding enabling support for tagged fields for the OffsetCommitValue
record, it would be great if you could give more details on the steps to
get there in the KIP. My understanding is that we would have to do the
following: 1) Update the code which reads the records to fail back to the
highest known version if the version stored in the log is unknown. Let's
say that we do this in AK 3.5. 2) We need to turn on tagged fields for the
record. I think that we can only do this in AK 3.6+.

4. I may have missed this part but we should clearly explain the drawback
of the proposed approach as well. Say that we enable tagged fields for
OffsetCommitValue in AK 3.6. This means that it won't be possible to
downgrade a cluster from 3.6 to a version earlier than 3.5. This is a
significant limitation in my opinion because, I think, users don't
necessarily upgrade to all versions.

5. In the proposal, it is not clear about whether the old software will
delete unknown records or not. It is true that new records will be deleted
when the group is downgraded but this only works if the operator respects
the process.

6. It would be great if we could extend the rejected alternative. The
alternative sounds clearly better when you read it so we should really
explain the reason to reject it. 1) One issue that you mention is that the
log must be compacted before downgrading and we don't really control this
process. 2) Transactions may be difficult to handle. I suppose that it is
possible to handle them though. Have you thought about this?

7. For the new dynamic configs, what happens if they are kept and the
quorum controller is downgraded?

Best,
David

On Thu, Mar 16, 2023 at 12:56 AM Jeff Kim 
wrote:

> Hi folks,
>
> I would like to start a discussion thread for KIP-915: Next Gen Group
> Coordinator Downgrade Path which proposes the downgrade design for the new
> group coordinator introduced in KIP-848
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-848%3A+The+Next+Generation+of+the+Consumer+Rebalance+Protocol
> >
> .
>
> KIP:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-915%3A+Next+Gen+Group+Coordinator+Downgrade+Path
>
> Thanks,
> Jeff
>

[DISCUSS] KIP-915: Next Gen Group Coordinator Downgrade Path

[Jeff Kim]

Hi folks,

I would like to start a discussion thread for KIP-915: Next Gen Group
Coordinator Downgrade Path which proposes the downgrade design for the new
group coordinator introduced in KIP-848

.

KIP:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-915%3A+Next+Gen+Group+Coordinator+Downgrade+Path

Thanks,
Jeff