Hi David, Thanks for bringing the KAFKA-15441 to this discussion thread, it seems to be a blocker and updated the JIRA. What is the plan to address that?
https://issues.apache.org/jira/browse/KAFKA-15411, which is about KRaft and delegation token related test failures. It seems to be a test env issue but it is better to check and confirm it is a non-blocker for 3.6.0. Please take a look and update the JIRA. You can add a special section in the release note about KRaft migrations being GA. Thanks, Satish. On Wed, 6 Sept 2023 at 19:52, David Arthur <mum...@gmail.com> wrote: > > Thanks, Satish! Here's another blocker > https://issues.apache.org/jira/browse/KAFKA-15441 :) > > For the 3.6 release notes and announcement, I'd like to include a special > note about ZK to KRaft migrations being GA (Generally Available). We have > finished closing all the gaps from the earlier releases of ZK migrations > (e.g., ACLs, SCRAM), so it is now possible to migrate all metadata to > KRaft. We have also made the migration more reliable and fault > tolerant with the inclusion of KIP-868 transactions. I'd be happy to write > something for the release notes when the time comes, if it's helpful. > > Thanks! > David > > On Tue, Sep 5, 2023 at 8:13 PM Satish Duggana <satish.dugg...@gmail.com> > wrote: > > > Hi David, > > Thanks for bringing this issue to this thread. > > I marked https://issues.apache.org/jira/browse/KAFKA-15435 as a blocker. > > > > Thanks, > > Satish. > > > > On Tue, 5 Sept 2023 at 21:29, David Arthur <mum...@gmail.com> wrote: > > > > > > Hi Satish. Thanks for running the release! > > > > > > I'd like to raise this as a blocker for 3.6 > > > https://issues.apache.org/jira/browse/KAFKA-15435. > > > > > > It's a very quick fix, so I should be able to post a PR soon. > > > > > > Thanks! > > > David > > > > > > On Mon, Sep 4, 2023 at 11:44 PM Justine Olshan > > <jols...@confluent.io.invalid> > > > wrote: > > > > > > > Thanks Satish. This is done 👍 > > > > > > > > Justine > > > > > > > > On Mon, Sep 4, 2023 at 5:16 PM Satish Duggana < > > satish.dugg...@gmail.com> > > > > wrote: > > > > > > > > > Hey Justine, > > > > > I went through KAFKA-15424 and the PR[1]. It seems there are no > > > > > dependent changes missing in 3.6 branch. They seem to be low risk as > > > > > you mentioned. Please merge it to the 3.6 branch as well. > > > > > > > > > > 1. https://github.com/apache/kafka/pull/14324. > > > > > > > > > > Thanks, > > > > > Satish. > > > > > > > > > > On Tue, 5 Sept 2023 at 05:06, Justine Olshan > > > > > <jols...@confluent.io.invalid> wrote: > > > > > > > > > > > > Sorry I meant to add the jira as well. > > > > > > https://issues.apache.org/jira/browse/KAFKA-15424 > > > > > > > > > > > > Justine > > > > > > > > > > > > On Mon, Sep 4, 2023 at 4:34 PM Justine Olshan < > > jols...@confluent.io> > > > > > wrote: > > > > > > > > > > > > > Hey Satish, > > > > > > > > > > > > > > I was working on adding dynamic configuration for > > > > > > > transaction verification. The PR is approved and ready to merge > > into > > > > > trunk. > > > > > > > I was thinking I could also add it to 3.6 since it is fairly low > > > > risk. > > > > > > > What do you think? > > > > > > > > > > > > > > Justine > > > > > > > > > > > > > > On Sat, Sep 2, 2023 at 6:21 PM Sophie Blee-Goldman < > > > > > ableegold...@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > >> Thanks Satish! The fix has been merged and cherrypicked to 3.6 > > > > > > >> > > > > > > >> On Sat, Sep 2, 2023 at 6:02 AM Satish Duggana < > > > > > satish.dugg...@gmail.com> > > > > > > >> wrote: > > > > > > >> > > > > > > >> > Hi Sophie, > > > > > > >> > Please feel free to add that to 3.6 branch as you say this is > > a > > > > > minor > > > > > > >> > change and will not cause any regressions. > > > > > > >> > > > > > > > >> > Thanks, > > > > > > >> > Satish. > > > > > > >> > > > > > > > >> > On Sat, 2 Sept 2023 at 08:44, Sophie Blee-Goldman > > > > > > >> > <ableegold...@gmail.com> wrote: > > > > > > >> > > > > > > > > >> > > Hey Satish, someone reported a minor bug in the Streams > > > > > application > > > > > > >> > > shutdown which was a recent regression, though not strictly > > a > > > > new > > > > > one: > > > > > > >> > was > > > > > > >> > > introduced in 3.4 I believe. > > > > > > >> > > > > > > > > >> > > The fix seems to be super lightweight and low-risk so I was > > > > > hoping to > > > > > > >> > slip > > > > > > >> > > it into 3.6 if that's ok with you? They plan to have the > > patch > > > > > > >> tonight. > > > > > > >> > > > > > > > > >> > > https://issues.apache.org/jira/browse/KAFKA-15429 > > > > > > >> > > > > > > > > >> > > On Thu, Aug 31, 2023 at 5:45 PM Satish Duggana < > > > > > > >> satish.dugg...@gmail.com > > > > > > >> > > > > > > > > >> > > wrote: > > > > > > >> > > > > > > > > >> > > > Thanks Chris for bringing this issue here and filing the > > new > > > > > JIRA > > > > > > >> for > > > > > > >> > > > 3.6.0[1]. It seems to be a blocker for 3.6.0. > > > > > > >> > > > > > > > > > >> > > > Please help review > > https://github.com/apache/kafka/pull/14314 > > > > > as > > > > > > >> Chris > > > > > > >> > > > requested. > > > > > > >> > > > > > > > > > >> > > > 1. https://issues.apache.org/jira/browse/KAFKA-15425 > > > > > > >> > > > > > > > > > >> > > > ~Satish. > > > > > > >> > > > > > > > > > >> > > > On Fri, 1 Sept 2023 at 03:59, Chris Egerton > > > > > <chr...@aiven.io.invalid > > > > > > >> > > > > > > > >> > > > wrote: > > > > > > >> > > > > > > > > > > >> > > > > Hi all, > > > > > > >> > > > > > > > > > > >> > > > > Quick update: I've filed a separate ticket, > > > > > > >> > > > > https://issues.apache.org/jira/browse/KAFKA-15425, to > > track > > > > > the > > > > > > >> > behavior > > > > > > >> > > > > change in Admin::listOffsets. For the full history of > > the > > > > > ticket, > > > > > > >> > it's > > > > > > >> > > > > worth reading the comment thread on the old ticket at > > > > > > >> > > > > https://issues.apache.org/jira/browse/KAFKA-12879. > > > > > > >> > > > > > > > > > > >> > > > > I've also published > > > > > https://github.com/apache/kafka/pull/14314 > > > > > > >> as a > > > > > > >> > > > fairly > > > > > > >> > > > > lightweight PR to revert the behavior of > > Admin::listOffsets > > > > > > >> without > > > > > > >> > also > > > > > > >> > > > > reverting the refactoring to use the internal admin > > driver > > > > > API. > > > > > > >> Would > > > > > > >> > > > > appreciate a review on that if anyone can spare the > > cycles. > > > > > > >> > > > > > > > > > > >> > > > > Cheers, > > > > > > >> > > > > > > > > > > >> > > > > Chris > > > > > > >> > > > > > > > > > > >> > > > > On Wed, Aug 30, 2023 at 1:01 PM Chris Egerton < > > > > > chr...@aiven.io> > > > > > > >> > wrote: > > > > > > >> > > > > > > > > > > >> > > > > > Hi Satish, > > > > > > >> > > > > > > > > > > > >> > > > > > Wanted to let you know that KAFKA-12879 ( > > > > > > >> > > > > > https://issues.apache.org/jira/browse/KAFKA-12879), a > > > > > breaking > > > > > > >> > change > > > > > > >> > > > in > > > > > > >> > > > > > Admin::listOffsets, has been reintroduced into the > > code > > > > > base. > > > > > > >> > Since we > > > > > > >> > > > > > haven't yet published a release with this change (at > > > > least, > > > > > not > > > > > > >> the > > > > > > >> > > > more > > > > > > >> > > > > > recent instance of it), I was hoping we could treat > > it as > > > > a > > > > > > >> > blocker for > > > > > > >> > > > > > 3.6.0. I'd also like to solicit the input of people > > > > familiar > > > > > > >> with > > > > > > >> > the > > > > > > >> > > > admin > > > > > > >> > > > > > client to weigh in on the Jira ticket about whether we > > > > > should > > > > > > >> > continue > > > > > > >> > > > to > > > > > > >> > > > > > preserve the current behavior (if the consensus is > > that we > > > > > > >> should, > > > > > > >> > I'm > > > > > > >> > > > > > happy to file a fix). > > > > > > >> > > > > > > > > > > > >> > > > > > Please let me know if you agree that this qualifies > > as a > > > > > > >> blocker. I > > > > > > >> > > > plan > > > > > > >> > > > > > on publishing a potential fix sometime this week. > > > > > > >> > > > > > > > > > > > >> > > > > > Cheers, > > > > > > >> > > > > > > > > > > > >> > > > > > Chris > > > > > > >> > > > > > > > > > > > >> > > > > > On Wed, Aug 30, 2023 at 9:19 AM Satish Duggana < > > > > > > >> > > > satish.dugg...@gmail.com> > > > > > > >> > > > > > wrote: > > > > > > >> > > > > > > > > > > > >> > > > > >> Hi, > > > > > > >> > > > > >> Please plan to continue merging pull requests > > associated > > > > > with > > > > > > >> any > > > > > > >> > > > > >> outstanding minor features and stabilization changes > > to > > > > 3.6 > > > > > > >> branch > > > > > > >> > > > > >> before September 3rd. Kindly update the KIP's > > > > > implementation > > > > > > >> > status in > > > > > > >> > > > > >> the 3.6.0 release notes. > > > > > > >> > > > > >> > > > > > > >> > > > > >> Thanks, > > > > > > >> > > > > >> Satish. > > > > > > >> > > > > >> > > > > > > >> > > > > >> On Fri, 25 Aug 2023 at 21:37, Justine Olshan > > > > > > >> > > > > >> <jols...@confluent.io.invalid> wrote: > > > > > > >> > > > > >> > > > > > > > >> > > > > >> > Hey Satish, > > > > > > >> > > > > >> > Everything should be in 3.6, and I will update the > > > > > release > > > > > > >> plan > > > > > > >> > > > wiki. > > > > > > >> > > > > >> > Thanks! > > > > > > >> > > > > >> > > > > > > > >> > > > > >> > On Fri, Aug 25, 2023 at 4:08 AM Satish Duggana < > > > > > > >> > > > > >> satish.dugg...@gmail.com> > > > > > > >> > > > > >> > wrote: > > > > > > >> > > > > >> > > > > > > > >> > > > > >> > > Hi Justine, > > > > > > >> > > > > >> > > Adding KIP-890 part-1 to 3.6.0 seems reasonable > > to > > > > me. > > > > > This > > > > > > >> > part > > > > > > >> > > > looks > > > > > > >> > > > > >> > > to be addressing a critical issue of consumers > > > > getting > > > > > > >> stuck. > > > > > > >> > > > Please > > > > > > >> > > > > >> > > update the release plan wiki and merge all the > > > > required > > > > > > >> > changes > > > > > > >> > > > to 3.6 > > > > > > >> > > > > >> > > branch. > > > > > > >> > > > > >> > > > > > > > > >> > > > > >> > > Thanks, > > > > > > >> > > > > >> > > Satish. > > > > > > >> > > > > >> > > > > > > > > >> > > > > >> > > On Thu, 24 Aug 2023 at 22:19, Justine Olshan > > > > > > >> > > > > >> > > <jols...@confluent.io.invalid> wrote: > > > > > > >> > > > > >> > > > > > > > > > >> > > > > >> > > > Hey Satish, > > > > > > >> > > > > >> > > > Does it make sense to include KIP-890 part 1? > > It > > > > > prevents > > > > > > >> > > > hanging > > > > > > >> > > > > >> > > > transactions for older clients. (An > > optimization > > > > and > > > > > > >> > stronger > > > > > > >> > > > EOS > > > > > > >> > > > > >> > > > guarantees will be included in part 2) > > > > > > >> > > > > >> > > > > > > > > > >> > > > > >> > > > Thanks, > > > > > > >> > > > > >> > > > Justine > > > > > > >> > > > > >> > > > > > > > > > >> > > > > >> > > > On Mon, Aug 21, 2023 at 3:29 AM Satish Duggana > > < > > > > > > >> > > > > >> satish.dugg...@gmail.com > > > > > > >> > > > > >> > > > > > > > > > >> > > > > >> > > > wrote: > > > > > > >> > > > > >> > > > > > > > > > >> > > > > >> > > > > Hi, > > > > > > >> > > > > >> > > > > 3.6 branch is created. Please make sure any > > PRs > > > > > > >> targeted > > > > > > >> > for > > > > > > >> > > > 3.6.0 > > > > > > >> > > > > >> > > > > should be merged to 3.6 branch once those are > > > > > merged to > > > > > > >> > trunk. > > > > > > >> > > > > >> > > > > > > > > > > >> > > > > >> > > > > Thanks, > > > > > > >> > > > > >> > > > > Satish. > > > > > > >> > > > > >> > > > > > > > > > > >> > > > > >> > > > > On Wed, 16 Aug 2023 at 15:58, Satish Duggana > > < > > > > > > >> > > > > >> satish.dugg...@gmail.com > > > > > > >> > > > > >> > > > > > > > > > >> > > > > >> > > > > wrote: > > > > > > >> > > > > >> > > > > > > > > > > > >> > > > > >> > > > > > Hi, > > > > > > >> > > > > >> > > > > > Please plan to merge PRs(including the > > major > > > > > > >> features) > > > > > > >> > > > targeted > > > > > > >> > > > > >> for > > > > > > >> > > > > >> > > > > > 3.6.0 by the end of Aug 20th UTC. Starting > > from > > > > > > >> August > > > > > > >> > 21st, > > > > > > >> > > > > >> any pull > > > > > > >> > > > > >> > > > > > requests intended for the 3.6.0 release > > must > > > > > include > > > > > > >> the > > > > > > >> > > > changes > > > > > > >> > > > > >> > > > > > merged into the 3.6 branch as mentioned in > > the > > > > > > >> release > > > > > > >> > plan. > > > > > > >> > > > > >> > > > > > > > > > > > >> > > > > >> > > > > > Thanks, > > > > > > >> > > > > >> > > > > > Satish. > > > > > > >> > > > > >> > > > > > > > > > > > >> > > > > >> > > > > > On Fri, 4 Aug 2023 at 18:39, Chris Egerton > > > > > > >> > > > > >> <chr...@aiven.io.invalid> > > > > > > >> > > > > >> > > > > wrote: > > > > > > >> > > > > >> > > > > > > > > > > > > >> > > > > >> > > > > > > Thanks for adding KIP-949, Satish! > > > > > > >> > > > > >> > > > > > > > > > > > > >> > > > > >> > > > > > > On Fri, Aug 4, 2023 at 7:06 AM Satish > > > > Duggana < > > > > > > >> > > > > >> > > > > satish.dugg...@gmail.com> > > > > > > >> > > > > >> > > > > > > wrote: > > > > > > >> > > > > >> > > > > > > > > > > > > >> > > > > >> > > > > > > > Hi, > > > > > > >> > > > > >> > > > > > > > Myself and Divij discussed and added > > the > > > > > wiki for > > > > > > >> > Kafka > > > > > > >> > > > > >> > > TieredStorage > > > > > > >> > > > > >> > > > > > > > Early Access Release[1]. If you have > > any > > > > > > >> comments or > > > > > > >> > > > > >> feedback, > > > > > > >> > > > > >> > > please > > > > > > >> > > > > >> > > > > > > > feel free to share them. > > > > > > >> > > > > >> > > > > > > > > > > > > > >> > > > > >> > > > > > > > 1. > > > > > > >> > > > > >> > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Tiered+Storage+Early+Access+Release+Notes > > > > > > >> > > > > >> > > > > > > > > > > > > > >> > > > > >> > > > > > > > Thanks, > > > > > > >> > > > > >> > > > > > > > Satish. > > > > > > >> > > > > >> > > > > > > > > > > > > > >> > > > > >> > > > > > > > On Fri, 4 Aug 2023 at 08:40, Satish > > > > Duggana < > > > > > > >> > > > > >> > > > > satish.dugg...@gmail.com> > > > > > > >> > > > > >> > > > > > > > wrote: > > > > > > >> > > > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > Hi Chris, > > > > > > >> > > > > >> > > > > > > > > Thanks for the update. This looks to > > be a > > > > > minor > > > > > > >> > change > > > > > > >> > > > > >> and is > > > > > > >> > > > > >> > > also > > > > > > >> > > > > >> > > > > > > > > useful for backward compatibility. I > > > > added > > > > > it > > > > > > >> to > > > > > > >> > the > > > > > > >> > > > > >> release > > > > > > >> > > > > >> > > plan > > > > > > >> > > > > >> > > > > as > > > > > > >> > > > > >> > > > > > > > > an exceptional case. > > > > > > >> > > > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > ~Satish. > > > > > > >> > > > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > On Thu, 3 Aug 2023 at 21:34, Chris > > > > Egerton > > > > > > >> > > > > >> > > <chr...@aiven.io.invalid > > > > > > >> > > > > >> > > > > > > > > > > > >> > > > > >> > > > > > > > wrote: > > > > > > >> > > > > >> > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > Hi Satish, > > > > > > >> > > > > >> > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > Would it be possible to include > > > > KIP-949 ( > > > > > > >> > > > > >> > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-949%3A+Add+flag+to+enable+the+usage+of+topic+separator+in+MM2+DefaultReplicationPolicy > > > > > > >> > > > > >> > > > > > > > ) > > > > > > >> > > > > >> > > > > > > > > > in the 3.6.0 release? It passed > > voting > > > > > > >> > yesterday, > > > > > > >> > > > and > > > > > > >> > > > > >> is a > > > > > > >> > > > > >> > > very > > > > > > >> > > > > >> > > > > small, > > > > > > >> > > > > >> > > > > > > > > > low-risk change that we'd like to > > put > > > > > out as > > > > > > >> > soon as > > > > > > >> > > > > >> > > possible in > > > > > > >> > > > > >> > > > > order > > > > > > >> > > > > >> > > > > > > > to > > > > > > >> > > > > >> > > > > > > > > > patch an accidental break in > > backwards > > > > > > >> > compatibility > > > > > > >> > > > > >> caused > > > > > > >> > > > > >> > > a few > > > > > > >> > > > > >> > > > > > > > versions > > > > > > >> > > > > >> > > > > > > > > > ago. > > > > > > >> > > > > >> > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > Best, > > > > > > >> > > > > >> > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > Chris > > > > > > >> > > > > >> > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > On Fri, Jul 28, 2023 at 2:35 AM > > Satish > > > > > > >> Duggana < > > > > > > >> > > > > >> > > > > > > > satish.dugg...@gmail.com> > > > > > > >> > > > > >> > > > > > > > > > wrote: > > > > > > >> > > > > >> > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > Hi All, > > > > > > >> > > > > >> > > > > > > > > > > Whoever has KIP entries in the > > 3.6.0 > > > > > > >> release > > > > > > >> > plan. > > > > > > >> > > > > >> Please > > > > > > >> > > > > >> > > > > update it > > > > > > >> > > > > >> > > > > > > > > > > with the latest status by > > > > tomorrow(end > > > > > of > > > > > > >> the > > > > > > >> > day > > > > > > >> > > > > >> 29th Jul > > > > > > >> > > > > >> > > UTC > > > > > > >> > > > > >> > > > > ). > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > Thanks > > > > > > >> > > > > >> > > > > > > > > > > Satish. > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > On Fri, 28 Jul 2023 at 12:01, > > Satish > > > > > > >> Duggana < > > > > > > >> > > > > >> > > > > > > > satish.dugg...@gmail.com> > > > > > > >> > > > > >> > > > > > > > > > > wrote: > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > Thanks Ismael and Divij for the > > > > > > >> suggestions. > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > One way was to follow the > > earlier > > > > > > >> guidelines > > > > > > >> > > > that > > > > > > >> > > > > >> we set > > > > > > >> > > > > >> > > for > > > > > > >> > > > > >> > > > > any > > > > > > >> > > > > >> > > > > > > > early > > > > > > >> > > > > >> > > > > > > > > > > > access release. It looks Ismael > > > > > already > > > > > > >> > > > mentioned > > > > > > >> > > > > >> the > > > > > > >> > > > > >> > > > > example of > > > > > > >> > > > > >> > > > > > > > > > > > KRaft. > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > KIP-405 mentions > > upgrade/downgrade > > > > > and > > > > > > >> > > > limitations > > > > > > >> > > > > >> > > sections. > > > > > > >> > > > > >> > > > > We can > > > > > > >> > > > > >> > > > > > > > > > > > clarify that in the release > > notes > > > > for > > > > > > >> users > > > > > > >> > on > > > > > > >> > > > how > > > > > > >> > > > > >> this > > > > > > >> > > > > >> > > > > feature > > > > > > >> > > > > >> > > > > > > > can be > > > > > > >> > > > > >> > > > > > > > > > > > used for early access. > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > Divij, We do not want users to > > > > enable > > > > > > >> this > > > > > > >> > > > feature > > > > > > >> > > > > >> on > > > > > > >> > > > > >> > > > > production > > > > > > >> > > > > >> > > > > > > > > > > > environments in early access > > > > release. > > > > > > >> Let us > > > > > > >> > > > work > > > > > > >> > > > > >> > > together > > > > > > >> > > > > >> > > > > on the > > > > > > >> > > > > >> > > > > > > > > > > > followups Ismael suggested. > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > ~Satish. > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > On Fri, 28 Jul 2023 at 02:24, > > Divij > > > > > > >> Vaidya < > > > > > > >> > > > > >> > > > > > > > divijvaidy...@gmail.com> > > > > > > >> > > > > >> > > > > > > > > > > wrote: > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > Those are great suggestions, > > > > thank > > > > > > >> you. We > > > > > > >> > > > will > > > > > > >> > > > > >> > > continue > > > > > > >> > > > > >> > > > > this > > > > > > >> > > > > >> > > > > > > > > > > discussion > > > > > > >> > > > > >> > > > > > > > > > > > > forward in a separate KIP for > > > > > release > > > > > > >> > plan for > > > > > > >> > > > > >> Tiered > > > > > > >> > > > > >> > > > > Storage. > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > On Thu 27. Jul 2023 at 21:46, > > > > > Ismael > > > > > > >> Juma > > > > > > >> > < > > > > > > >> > > > > >> > > > > m...@ismaeljuma.com> > > > > > > >> > > > > >> > > > > > > > wrote: > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > Hi Divij, > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > I think the points you > > bring up > > > > > for > > > > > > >> > > > discussion > > > > > > >> > > > > >> are > > > > > > >> > > > > >> > > all > > > > > > >> > > > > >> > > > > good. > > > > > > >> > > > > >> > > > > > > > My main > > > > > > >> > > > > >> > > > > > > > > > > > > > feedback is that they > > should be > > > > > > >> > discussed > > > > > > >> > > > in the > > > > > > >> > > > > >> > > context > > > > > > >> > > > > >> > > > > of > > > > > > >> > > > > >> > > > > > > > KIPs vs > > > > > > >> > > > > >> > > > > > > > > > > the > > > > > > >> > > > > >> > > > > > > > > > > > > > release template. That's > > why we > > > > > have > > > > > > >> a > > > > > > >> > > > backwards > > > > > > >> > > > > >> > > > > compatibility > > > > > > >> > > > > >> > > > > > > > > > > section for > > > > > > >> > > > > >> > > > > > > > > > > > > > every KIP, it's precisely > > to > > > > > ensure > > > > > > >> we > > > > > > >> > think > > > > > > >> > > > > >> > > carefully > > > > > > >> > > > > >> > > > > about > > > > > > >> > > > > >> > > > > > > > some of > > > > > > >> > > > > >> > > > > > > > > > > the > > > > > > >> > > > > >> > > > > > > > > > > > > > points you're bringing up. > > When > > > > > it > > > > > > >> > comes to > > > > > > >> > > > > >> defining > > > > > > >> > > > > >> > > the > > > > > > >> > > > > >> > > > > > > > meaning of > > > > > > >> > > > > >> > > > > > > > > > > early > > > > > > >> > > > > >> > > > > > > > > > > > > > access, we have two > > options: > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > 1. Have a KIP specifically > > for > > > > > tiered > > > > > > >> > > > storage. > > > > > > >> > > > > >> > > > > > > > > > > > > > 2. Have a KIP to define > > general > > > > > > >> > guidelines > > > > > > >> > > > for > > > > > > >> > > > > >> what > > > > > > >> > > > > >> > > early > > > > > > >> > > > > >> > > > > > > > access > > > > > > >> > > > > >> > > > > > > > > > > means. > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > Does this make sense? > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > Ismael > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > On Thu, Jul 27, 2023 at > > 6:38 PM > > > > > Divij > > > > > > >> > > > Vaidya < > > > > > > >> > > > > >> > > > > > > > > > > divijvaidy...@gmail.com> > > > > > > >> > > > > >> > > > > > > > > > > > > > wrote: > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > Thank you for the > > response, > > > > > Ismael. > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > 1. Specifically in > > context of > > > > > 3.6, > > > > > > >> I > > > > > > >> > > > wanted > > > > > > >> > > > > >> this > > > > > > >> > > > > >> > > > > > > > compatibility > > > > > > >> > > > > >> > > > > > > > > > > > > > > guarantee point to > > encourage > > > > a > > > > > > >> > discussion > > > > > > >> > > > on > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-952%3A+Regenerate+segment-aligned+producer+snapshots+when+upgrading+to+a+Kafka+version+supporting+Tiered+Storage > > > > > > >> > > > > >> > > > > > > > > > > > > > > . > > > > > > >> > > > > >> > > > > > > > > > > > > > > Due to lack of producer > > > > > snapshots > > > > > > >> in > > > > > > >> > <2.8 > > > > > > >> > > > > >> > > versions, a > > > > > > >> > > > > >> > > > > > > > customer may > > > > > > >> > > > > >> > > > > > > > > > > not > > > > > > >> > > > > >> > > > > > > > > > > > > > > be able to upgrade to > > 3.6 and > > > > > use > > > > > > >> TS > > > > > > >> > on a > > > > > > >> > > > > >> topic > > > > > > >> > > > > >> > > which > > > > > > >> > > > > >> > > > > was > > > > > > >> > > > > >> > > > > > > > created > > > > > > >> > > > > >> > > > > > > > > > > when > > > > > > >> > > > > >> > > > > > > > > > > > > > > the cluster was on <2.8 > > > > version > > > > > > >> (see > > > > > > >> > > > > >> motivation for > > > > > > >> > > > > >> > > > > > > > details). We > > > > > > >> > > > > >> > > > > > > > > > > can > > > > > > >> > > > > >> > > > > > > > > > > > > > > discuss and agree that it > > > > does > > > > > not > > > > > > >> > break > > > > > > >> > > > > >> > > compatibility, > > > > > > >> > > > > >> > > > > > > > which is > > > > > > >> > > > > >> > > > > > > > > > > fine. > > > > > > >> > > > > >> > > > > > > > > > > > > > > But I want to ensure > > that we > > > > > have a > > > > > > >> > > > discussion > > > > > > >> > > > > >> > > soon on > > > > > > >> > > > > >> > > > > this > > > > > > >> > > > > >> > > > > > > > to > > > > > > >> > > > > >> > > > > > > > > > > reach a > > > > > > >> > > > > >> > > > > > > > > > > > > > > conclusion. > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > 2. I will start a KIP on > > this > > > > > for > > > > > > >> > further > > > > > > >> > > > > >> > > discussion. > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > 3. In the context of 3.6, > > > > this > > > > > > >> would > > > > > > >> > mean > > > > > > >> > > > that > > > > > > >> > > > > >> > > there > > > > > > >> > > > > >> > > > > should > > > > > > >> > > > > >> > > > > > > > be > > > > > > >> > > > > >> > > > > > > > > > > > > > > no-regression, if a user > > does > > > > > "not" > > > > > > >> > > > turn-on > > > > > > >> > > > > >> remote > > > > > > >> > > > > >> > > > > storage > > > > > > >> > > > > >> > > > > > > > (early > > > > > > >> > > > > >> > > > > > > > > > > > > > > access feature) at a > > cluster > > > > > > >> level. We > > > > > > >> > > > have > > > > > > >> > > > > >> some > > > > > > >> > > > > >> > > known > > > > > > >> > > > > >> > > > > cases > > > > > > >> > > > > >> > > > > > > > (such > > > > > > >> > > > > >> > > > > > > > > > > as > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > >> https://issues.apache.org/jira/browse/KAFKA-15189) > > > > > > >> > > > > >> > > > > which > > > > > > >> > > > > >> > > > > > > > violate > > > > > > >> > > > > >> > > > > > > > > > > this > > > > > > >> > > > > >> > > > > > > > > > > > > > > compatibility > > requirement. > > > > > Having > > > > > > >> this > > > > > > >> > > > > >> guarantee > > > > > > >> > > > > >> > > > > mentioned > > > > > > >> > > > > >> > > > > > > > in the > > > > > > >> > > > > >> > > > > > > > > > > > > > > release plan will ensure > > that > > > > > we > > > > > > >> are > > > > > > >> > all > > > > > > >> > > > in > > > > > > >> > > > > >> > > agreement > > > > > > >> > > > > >> > > > > with > > > > > > >> > > > > >> > > > > > > > which > > > > > > >> > > > > >> > > > > > > > > > > cases > > > > > > >> > > > > >> > > > > > > > > > > > > > > are truly blockers and > > which > > > > > > >> aren't. > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > 4. Fair, instead of a > > general > > > > > goal, > > > > > > >> > let me > > > > > > >> > > > > >> put it > > > > > > >> > > > > >> > > > > > > > specifically in > > > > > > >> > > > > >> > > > > > > > > > > the > > > > > > >> > > > > >> > > > > > > > > > > > > > > context of 3.6. Let me > > know > > > > if > > > > > > >> this is > > > > > > >> > > > not the > > > > > > >> > > > > >> > > right > > > > > > >> > > > > >> > > > > forum > > > > > > >> > > > > >> > > > > > > > for this > > > > > > >> > > > > >> > > > > > > > > > > > > > > discussion. > > > > > > >> > > > > >> > > > > > > > > > > > > > > Once a user "turns on" > > tiered > > > > > > >> storage > > > > > > >> > > > (TS) at > > > > > > >> > > > > >> a > > > > > > >> > > > > >> > > cluster > > > > > > >> > > > > >> > > > > > > > level, I am > > > > > > >> > > > > >> > > > > > > > > > > > > > > proposing that they > > should > > > > > have the > > > > > > >> > > > ability to > > > > > > >> > > > > >> > > turn it > > > > > > >> > > > > >> > > > > off > > > > > > >> > > > > >> > > > > > > > as well > > > > > > >> > > > > >> > > > > > > > > > > at > > > > > > >> > > > > >> > > > > > > > > > > > > > > a cluster level. Since > > this > > > > is > > > > > a > > > > > > >> topic > > > > > > >> > > > level > > > > > > >> > > > > >> > > feature, > > > > > > >> > > > > >> > > > > folks > > > > > > >> > > > > >> > > > > > > > may not > > > > > > >> > > > > >> > > > > > > > > > > > > > > spin up a separate > > cluster to > > > > > try > > > > > > >> this > > > > > > >> > > > > >> feature, > > > > > > >> > > > > >> > > hence, > > > > > > >> > > > > >> > > > > we > > > > > > >> > > > > >> > > > > > > > need to > > > > > > >> > > > > >> > > > > > > > > > > > > > > ensure that we provide > > them > > > > > with > > > > > > >> the > > > > > > >> > > > ability > > > > > > >> > > > > >> to try > > > > > > >> > > > > >> > > > > tiered > > > > > > >> > > > > >> > > > > > > > storage > > > > > > >> > > > > >> > > > > > > > > > > for > > > > > > >> > > > > >> > > > > > > > > > > > > > > a topic which could be > > > > deleted > > > > > and > > > > > > >> > > > featured > > > > > > >> > > > > >> > > > > turned-off, so > > > > > > >> > > > > >> > > > > > > > that > > > > > > >> > > > > >> > > > > > > > > > > rest > > > > > > >> > > > > >> > > > > > > > > > > > > > > of the production cases > > are > > > > not > > > > > > >> > impacted. > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > 5. Agree on not making > > public > > > > > > >> > interface > > > > > > >> > > > > >> change as a > > > > > > >> > > > > >> > > > > > > > requirement > > > > > > >> > > > > >> > > > > > > > > > > but we > > > > > > >> > > > > >> > > > > > > > > > > > > > > should define what "early > > > > > access" > > > > > > >> > means in > > > > > > >> > > > > >> that > > > > > > >> > > > > >> > > case. > > > > > > >> > > > > >> > > > > Users > > > > > > >> > > > > >> > > > > > > > may > > > > > > >> > > > > >> > > > > > > > > > > not be > > > > > > >> > > > > >> > > > > > > > > > > > > > > aware that "early access" > > > > > public > > > > > > >> APIs > > > > > > >> > may > > > > > > >> > > > > >> change > > > > > > >> > > > > >> > > > > (unless I am > > > > > > >> > > > > >> > > > > > > > > > > missing > > > > > > >> > > > > >> > > > > > > > > > > > > > > some documentation > > somewhere > > > > > > >> > completely, > > > > > > >> > > > in > > > > > > >> > > > > >> which > > > > > > >> > > > > >> > > case > > > > > > >> > > > > >> > > > > I > > > > > > >> > > > > >> > > > > > > > apologize > > > > > > >> > > > > >> > > > > > > > > > > for > > > > > > >> > > > > >> > > > > > > > > > > > > > > bringing this naive > > point). > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > -- > > > > > > >> > > > > >> > > > > > > > > > > > > > > Divij Vaidya > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > On Thu, Jul 27, 2023 at > > > > 2:27 PM > > > > > > >> Ismael > > > > > > >> > > > Juma < > > > > > > >> > > > > >> > > > > > > > m...@ismaeljuma.com> > > > > > > >> > > > > >> > > > > > > > > > > wrote: > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > Hi Divij, > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > Some of these are > > launch > > > > > > >> checklist > > > > > > >> > items > > > > > > >> > > > > >> (not > > > > > > >> > > > > >> > > really > > > > > > >> > > > > >> > > > > > > > goals) and > > > > > > >> > > > > >> > > > > > > > > > > some > > > > > > >> > > > > >> > > > > > > > > > > > > > are > > > > > > >> > > > > >> > > > > > > > > > > > > > > > compatibility > > guarantees. > > > > > More > > > > > > >> > below. > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > On Thu, Jul 27, 2023, > > 12:10 > > > > > PM > > > > > > >> Divij > > > > > > >> > > > Vaidya > > > > > > >> > > > > >> < > > > > > > >> > > > > >> > > > > > > > > > > divijvaidy...@gmail.com> > > > > > > >> > > > > >> > > > > > > > > > > > > > > wrote: > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > Hey Satish > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > Could we consider > > adding > > > > > > >> "launch > > > > > > >> > > > goals" > > > > > > >> > > > > >> in the > > > > > > >> > > > > >> > > > > release > > > > > > >> > > > > >> > > > > > > > plan. > > > > > > >> > > > > >> > > > > > > > > > > While > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > some of these may be > > > > > implicit, > > > > > > >> it > > > > > > >> > > > would be > > > > > > >> > > > > >> > > nice to > > > > > > >> > > > > >> > > > > list > > > > > > >> > > > > >> > > > > > > > them > > > > > > >> > > > > >> > > > > > > > > > > down in > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > the release plan. For > > > > this > > > > > > >> > release, > > > > > > >> > > > our > > > > > > >> > > > > >> launch > > > > > > >> > > > > >> > > > > > > > requirements > > > > > > >> > > > > >> > > > > > > > > > > would be: > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > 1. Users should be > > able > > > > to > > > > > > >> upgrade > > > > > > >> > > > from > > > > > > >> > > > > >> any > > > > > > >> > > > > >> > > prior > > > > > > >> > > > > >> > > > > Kafka > > > > > > >> > > > > >> > > > > > > > > > > version to > > > > > > >> > > > > >> > > > > > > > > > > > > > this > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > version. > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > This is part of the > > > > > compatibility > > > > > > >> > > > > >> guarantees. The > > > > > > >> > > > > >> > > > > upgrade > > > > > > >> > > > > >> > > > > > > > notes > > > > > > >> > > > > >> > > > > > > > > > > mention > > > > > > >> > > > > >> > > > > > > > > > > > > > > > this already. If there > > is a > > > > > > >> change > > > > > > >> > in a > > > > > > >> > > > > >> given > > > > > > >> > > > > >> > > > > release, it > > > > > > >> > > > > >> > > > > > > > should > > > > > > >> > > > > >> > > > > > > > > > > > > > > definitely > > > > > > >> > > > > >> > > > > > > > > > > > > > > > be highlighted. > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > 2. On release, this > > version > > > > > (or > > > > > > >> it's > > > > > > >> > > > > >> > > dependencies) > > > > > > >> > > > > >> > > > > would > > > > > > >> > > > > >> > > > > > > > not > > > > > > >> > > > > >> > > > > > > > > > > have any > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > known MEDIUM/HIGH > > CVE. > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > This is a new policy > > and > > > > the > > > > > > >> details > > > > > > >> > > > should > > > > > > >> > > > > >> be > > > > > > >> > > > > >> > > > > discussed. > > > > > > >> > > > > >> > > > > > > > In > > > > > > >> > > > > >> > > > > > > > > > > > > > particular, > > > > > > >> > > > > >> > > > > > > > > > > > > > > > the threshold (medium > > or > > > > > high). > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > 3. Presence of any > > "early > > > > > > >> > access"/"beta" > > > > > > >> > > > > >> feature > > > > > > >> > > > > >> > > > > should not > > > > > > >> > > > > >> > > > > > > > > > > impact > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > other production > > features > > > > > when > > > > > > >> it > > > > > > >> > is > > > > > > >> > > > not > > > > > > >> > > > > >> > > enabled. > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > This is a general > > guideline > > > > > for > > > > > > >> > early > > > > > > >> > > > access > > > > > > >> > > > > >> > > > > features and > > > > > > >> > > > > >> > > > > > > > not > > > > > > >> > > > > >> > > > > > > > > > > specific > > > > > > >> > > > > >> > > > > > > > > > > > > > to > > > > > > >> > > > > >> > > > > > > > > > > > > > > > this release. It would > > be > > > > > good to > > > > > > >> > have a > > > > > > >> > > > > >> page > > > > > > >> > > > > >> > > that > > > > > > >> > > > > >> > > > > talks > > > > > > >> > > > > >> > > > > > > > about > > > > > > >> > > > > >> > > > > > > > > > > these > > > > > > >> > > > > >> > > > > > > > > > > > > > > things. > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > 4. Once enabled, users > > > > should > > > > > > >> have > > > > > > >> > an > > > > > > >> > > > > >> option to > > > > > > >> > > > > >> > > > > disable any > > > > > > >> > > > > >> > > > > > > > > > > "early > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > access"/"beta" > > feature > > > > and > > > > > > >> resume > > > > > > >> > > > normal > > > > > > >> > > > > >> > > production > > > > > > >> > > > > >> > > > > > > > features, > > > > > > >> > > > > >> > > > > > > > > > > i.e. > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > impact of beta > > features > > > > > should > > > > > > >> be > > > > > > >> > > > > >> reversible. > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > This needs discussion > > and I > > > > > don't > > > > > > >> > think > > > > > > >> > > > it's > > > > > > >> > > > > >> > > > > reasonable as > > > > > > >> > > > > >> > > > > > > > a > > > > > > >> > > > > >> > > > > > > > > > > general > > > > > > >> > > > > >> > > > > > > > > > > > > > > rule. > > > > > > >> > > > > >> > > > > > > > > > > > > > > > For example, Kraft > > early > > > > > access > > > > > > >> > wasn't > > > > > > >> > > > > >> reversible > > > > > > >> > > > > >> > > > > and it > > > > > > >> > > > > >> > > > > > > > was not > > > > > > >> > > > > >> > > > > > > > > > > > > > feasible > > > > > > >> > > > > >> > > > > > > > > > > > > > > > for it to be. > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > 5. KIP-405 will be > > > > available > > > > > in > > > > > > >> > "early > > > > > > >> > > > > >> > > access"/"beta" > > > > > > >> > > > > >> > > > > > > > mode. Early > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > access/beta means > > that > > > > the > > > > > > >> public > > > > > > >> > > > facing > > > > > > >> > > > > >> > > > > interfaces won't > > > > > > >> > > > > >> > > > > > > > > > > change in > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > future but the > > > > > implementation > > > > > > >> is > > > > > > >> > not > > > > > > >> > > > > >> > > recommended > > > > > > >> > > > > >> > > > > to be > > > > > > >> > > > > >> > > > > > > > used in > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > production. > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > I don't think it's ok > > to > > > > make > > > > > > >> this a > > > > > > >> > > > > >> requirement. > > > > > > >> > > > > >> > > > > Early > > > > > > >> > > > > >> > > > > > > > access > > > > > > >> > > > > >> > > > > > > > > > > is a way > > > > > > >> > > > > >> > > > > > > > > > > > > > > to > > > > > > >> > > > > >> > > > > > > > > > > > > > > > get early feedback and > > all > > > > > types > > > > > > >> of > > > > > > >> > > > changes > > > > > > >> > > > > >> > > should > > > > > > >> > > > > >> > > > > be on > > > > > > >> > > > > >> > > > > > > > the > > > > > > >> > > > > >> > > > > > > > > > > table. > > > > > > >> > > > > >> > > > > > > > > > > > > > They > > > > > > >> > > > > >> > > > > > > > > > > > > > > > would be discussed via > > KIPs > > > > > as > > > > > > >> > usual. I > > > > > > >> > > > > >> believe > > > > > > >> > > > > >> > > > > there were > > > > > > >> > > > > >> > > > > > > > some > > > > > > >> > > > > >> > > > > > > > > > > > > > > > incompatible changes > > for > > > > > Kraft > > > > > > >> > during > > > > > > >> > > > the > > > > > > >> > > > > >> early > > > > > > >> > > > > >> > > > > access > > > > > > >> > > > > >> > > > > > > > period > > > > > > >> > > > > >> > > > > > > > > > > although > > > > > > >> > > > > >> > > > > > > > > > > > > > > the > > > > > > >> > > > > >> > > > > > > > > > > > > > > > team aimed to minimize > > work > > > > > > >> required > > > > > > >> > > > during > > > > > > >> > > > > >> > > > > upgrades. I > > > > > > >> > > > > >> > > > > > > > have > > > > > > >> > > > > >> > > > > > > > > > > mentioned > > > > > > >> > > > > >> > > > > > > > > > > > > > > > Kraft a couple of times > > > > since > > > > > > >> it's a > > > > > > >> > > > good > > > > > > >> > > > > >> > > example of > > > > > > >> > > > > >> > > > > a > > > > > > >> > > > > >> > > > > > > > large > > > > > > >> > > > > >> > > > > > > > > > > feature > > > > > > >> > > > > >> > > > > > > > > > > > > > that > > > > > > >> > > > > >> > > > > > > > > > > > > > > > went through this > > process. > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > Ismael > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > -- > > > > > > >> > > > > >> > > > > > > > > > > > > Divij Vaidya > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > David Arthur > > > > > > -- > David Arthur
