Thanks, Satish! Here's another blocker https://issues.apache.org/jira/browse/KAFKA-15441 :)
For the 3.6 release notes and announcement, I'd like to include a special note about ZK to KRaft migrations being GA (Generally Available). We have finished closing all the gaps from the earlier releases of ZK migrations (e.g., ACLs, SCRAM), so it is now possible to migrate all metadata to KRaft. We have also made the migration more reliable and fault tolerant with the inclusion of KIP-868 transactions. I'd be happy to write something for the release notes when the time comes, if it's helpful. Thanks! David On Tue, Sep 5, 2023 at 8:13 PM Satish Duggana <satish.dugg...@gmail.com> wrote: > Hi David, > Thanks for bringing this issue to this thread. > I marked https://issues.apache.org/jira/browse/KAFKA-15435 as a blocker. > > Thanks, > Satish. > > On Tue, 5 Sept 2023 at 21:29, David Arthur <mum...@gmail.com> wrote: > > > > Hi Satish. Thanks for running the release! > > > > I'd like to raise this as a blocker for 3.6 > > https://issues.apache.org/jira/browse/KAFKA-15435. > > > > It's a very quick fix, so I should be able to post a PR soon. > > > > Thanks! > > David > > > > On Mon, Sep 4, 2023 at 11:44 PM Justine Olshan > <jols...@confluent.io.invalid> > > wrote: > > > > > Thanks Satish. This is done 👍 > > > > > > Justine > > > > > > On Mon, Sep 4, 2023 at 5:16 PM Satish Duggana < > satish.dugg...@gmail.com> > > > wrote: > > > > > > > Hey Justine, > > > > I went through KAFKA-15424 and the PR[1]. It seems there are no > > > > dependent changes missing in 3.6 branch. They seem to be low risk as > > > > you mentioned. Please merge it to the 3.6 branch as well. > > > > > > > > 1. https://github.com/apache/kafka/pull/14324. > > > > > > > > Thanks, > > > > Satish. > > > > > > > > On Tue, 5 Sept 2023 at 05:06, Justine Olshan > > > > <jols...@confluent.io.invalid> wrote: > > > > > > > > > > Sorry I meant to add the jira as well. > > > > > https://issues.apache.org/jira/browse/KAFKA-15424 > > > > > > > > > > Justine > > > > > > > > > > On Mon, Sep 4, 2023 at 4:34 PM Justine Olshan < > jols...@confluent.io> > > > > wrote: > > > > > > > > > > > Hey Satish, > > > > > > > > > > > > I was working on adding dynamic configuration for > > > > > > transaction verification. The PR is approved and ready to merge > into > > > > trunk. > > > > > > I was thinking I could also add it to 3.6 since it is fairly low > > > risk. > > > > > > What do you think? > > > > > > > > > > > > Justine > > > > > > > > > > > > On Sat, Sep 2, 2023 at 6:21 PM Sophie Blee-Goldman < > > > > ableegold...@gmail.com> > > > > > > wrote: > > > > > > > > > > > >> Thanks Satish! The fix has been merged and cherrypicked to 3.6 > > > > > >> > > > > > >> On Sat, Sep 2, 2023 at 6:02 AM Satish Duggana < > > > > satish.dugg...@gmail.com> > > > > > >> wrote: > > > > > >> > > > > > >> > Hi Sophie, > > > > > >> > Please feel free to add that to 3.6 branch as you say this is > a > > > > minor > > > > > >> > change and will not cause any regressions. > > > > > >> > > > > > > >> > Thanks, > > > > > >> > Satish. > > > > > >> > > > > > > >> > On Sat, 2 Sept 2023 at 08:44, Sophie Blee-Goldman > > > > > >> > <ableegold...@gmail.com> wrote: > > > > > >> > > > > > > > >> > > Hey Satish, someone reported a minor bug in the Streams > > > > application > > > > > >> > > shutdown which was a recent regression, though not strictly > a > > > new > > > > one: > > > > > >> > was > > > > > >> > > introduced in 3.4 I believe. > > > > > >> > > > > > > > >> > > The fix seems to be super lightweight and low-risk so I was > > > > hoping to > > > > > >> > slip > > > > > >> > > it into 3.6 if that's ok with you? They plan to have the > patch > > > > > >> tonight. > > > > > >> > > > > > > > >> > > https://issues.apache.org/jira/browse/KAFKA-15429 > > > > > >> > > > > > > > >> > > On Thu, Aug 31, 2023 at 5:45 PM Satish Duggana < > > > > > >> satish.dugg...@gmail.com > > > > > >> > > > > > > > >> > > wrote: > > > > > >> > > > > > > > >> > > > Thanks Chris for bringing this issue here and filing the > new > > > > JIRA > > > > > >> for > > > > > >> > > > 3.6.0[1]. It seems to be a blocker for 3.6.0. > > > > > >> > > > > > > > > >> > > > Please help review > https://github.com/apache/kafka/pull/14314 > > > > as > > > > > >> Chris > > > > > >> > > > requested. > > > > > >> > > > > > > > > >> > > > 1. https://issues.apache.org/jira/browse/KAFKA-15425 > > > > > >> > > > > > > > > >> > > > ~Satish. > > > > > >> > > > > > > > > >> > > > On Fri, 1 Sept 2023 at 03:59, Chris Egerton > > > > <chr...@aiven.io.invalid > > > > > >> > > > > > > >> > > > wrote: > > > > > >> > > > > > > > > > >> > > > > Hi all, > > > > > >> > > > > > > > > > >> > > > > Quick update: I've filed a separate ticket, > > > > > >> > > > > https://issues.apache.org/jira/browse/KAFKA-15425, to > track > > > > the > > > > > >> > behavior > > > > > >> > > > > change in Admin::listOffsets. For the full history of > the > > > > ticket, > > > > > >> > it's > > > > > >> > > > > worth reading the comment thread on the old ticket at > > > > > >> > > > > https://issues.apache.org/jira/browse/KAFKA-12879. > > > > > >> > > > > > > > > > >> > > > > I've also published > > > > https://github.com/apache/kafka/pull/14314 > > > > > >> as a > > > > > >> > > > fairly > > > > > >> > > > > lightweight PR to revert the behavior of > Admin::listOffsets > > > > > >> without > > > > > >> > also > > > > > >> > > > > reverting the refactoring to use the internal admin > driver > > > > API. > > > > > >> Would > > > > > >> > > > > appreciate a review on that if anyone can spare the > cycles. > > > > > >> > > > > > > > > > >> > > > > Cheers, > > > > > >> > > > > > > > > > >> > > > > Chris > > > > > >> > > > > > > > > > >> > > > > On Wed, Aug 30, 2023 at 1:01 PM Chris Egerton < > > > > chr...@aiven.io> > > > > > >> > wrote: > > > > > >> > > > > > > > > > >> > > > > > Hi Satish, > > > > > >> > > > > > > > > > > >> > > > > > Wanted to let you know that KAFKA-12879 ( > > > > > >> > > > > > https://issues.apache.org/jira/browse/KAFKA-12879), a > > > > breaking > > > > > >> > change > > > > > >> > > > in > > > > > >> > > > > > Admin::listOffsets, has been reintroduced into the > code > > > > base. > > > > > >> > Since we > > > > > >> > > > > > haven't yet published a release with this change (at > > > least, > > > > not > > > > > >> the > > > > > >> > > > more > > > > > >> > > > > > recent instance of it), I was hoping we could treat > it as > > > a > > > > > >> > blocker for > > > > > >> > > > > > 3.6.0. I'd also like to solicit the input of people > > > familiar > > > > > >> with > > > > > >> > the > > > > > >> > > > admin > > > > > >> > > > > > client to weigh in on the Jira ticket about whether we > > > > should > > > > > >> > continue > > > > > >> > > > to > > > > > >> > > > > > preserve the current behavior (if the consensus is > that we > > > > > >> should, > > > > > >> > I'm > > > > > >> > > > > > happy to file a fix). > > > > > >> > > > > > > > > > > >> > > > > > Please let me know if you agree that this qualifies > as a > > > > > >> blocker. I > > > > > >> > > > plan > > > > > >> > > > > > on publishing a potential fix sometime this week. > > > > > >> > > > > > > > > > > >> > > > > > Cheers, > > > > > >> > > > > > > > > > > >> > > > > > Chris > > > > > >> > > > > > > > > > > >> > > > > > On Wed, Aug 30, 2023 at 9:19 AM Satish Duggana < > > > > > >> > > > satish.dugg...@gmail.com> > > > > > >> > > > > > wrote: > > > > > >> > > > > > > > > > > >> > > > > >> Hi, > > > > > >> > > > > >> Please plan to continue merging pull requests > associated > > > > with > > > > > >> any > > > > > >> > > > > >> outstanding minor features and stabilization changes > to > > > 3.6 > > > > > >> branch > > > > > >> > > > > >> before September 3rd. Kindly update the KIP's > > > > implementation > > > > > >> > status in > > > > > >> > > > > >> the 3.6.0 release notes. > > > > > >> > > > > >> > > > > > >> > > > > >> Thanks, > > > > > >> > > > > >> Satish. > > > > > >> > > > > >> > > > > > >> > > > > >> On Fri, 25 Aug 2023 at 21:37, Justine Olshan > > > > > >> > > > > >> <jols...@confluent.io.invalid> wrote: > > > > > >> > > > > >> > > > > > > >> > > > > >> > Hey Satish, > > > > > >> > > > > >> > Everything should be in 3.6, and I will update the > > > > release > > > > > >> plan > > > > > >> > > > wiki. > > > > > >> > > > > >> > Thanks! > > > > > >> > > > > >> > > > > > > >> > > > > >> > On Fri, Aug 25, 2023 at 4:08 AM Satish Duggana < > > > > > >> > > > > >> satish.dugg...@gmail.com> > > > > > >> > > > > >> > wrote: > > > > > >> > > > > >> > > > > > > >> > > > > >> > > Hi Justine, > > > > > >> > > > > >> > > Adding KIP-890 part-1 to 3.6.0 seems reasonable > to > > > me. > > > > This > > > > > >> > part > > > > > >> > > > looks > > > > > >> > > > > >> > > to be addressing a critical issue of consumers > > > getting > > > > > >> stuck. > > > > > >> > > > Please > > > > > >> > > > > >> > > update the release plan wiki and merge all the > > > required > > > > > >> > changes > > > > > >> > > > to 3.6 > > > > > >> > > > > >> > > branch. > > > > > >> > > > > >> > > > > > > > >> > > > > >> > > Thanks, > > > > > >> > > > > >> > > Satish. > > > > > >> > > > > >> > > > > > > > >> > > > > >> > > On Thu, 24 Aug 2023 at 22:19, Justine Olshan > > > > > >> > > > > >> > > <jols...@confluent.io.invalid> wrote: > > > > > >> > > > > >> > > > > > > > > >> > > > > >> > > > Hey Satish, > > > > > >> > > > > >> > > > Does it make sense to include KIP-890 part 1? > It > > > > prevents > > > > > >> > > > hanging > > > > > >> > > > > >> > > > transactions for older clients. (An > optimization > > > and > > > > > >> > stronger > > > > > >> > > > EOS > > > > > >> > > > > >> > > > guarantees will be included in part 2) > > > > > >> > > > > >> > > > > > > > > >> > > > > >> > > > Thanks, > > > > > >> > > > > >> > > > Justine > > > > > >> > > > > >> > > > > > > > > >> > > > > >> > > > On Mon, Aug 21, 2023 at 3:29 AM Satish Duggana > < > > > > > >> > > > > >> satish.dugg...@gmail.com > > > > > >> > > > > >> > > > > > > > > >> > > > > >> > > > wrote: > > > > > >> > > > > >> > > > > > > > > >> > > > > >> > > > > Hi, > > > > > >> > > > > >> > > > > 3.6 branch is created. Please make sure any > PRs > > > > > >> targeted > > > > > >> > for > > > > > >> > > > 3.6.0 > > > > > >> > > > > >> > > > > should be merged to 3.6 branch once those are > > > > merged to > > > > > >> > trunk. > > > > > >> > > > > >> > > > > > > > > > >> > > > > >> > > > > Thanks, > > > > > >> > > > > >> > > > > Satish. > > > > > >> > > > > >> > > > > > > > > > >> > > > > >> > > > > On Wed, 16 Aug 2023 at 15:58, Satish Duggana > < > > > > > >> > > > > >> satish.dugg...@gmail.com > > > > > >> > > > > >> > > > > > > > > >> > > > > >> > > > > wrote: > > > > > >> > > > > >> > > > > > > > > > > >> > > > > >> > > > > > Hi, > > > > > >> > > > > >> > > > > > Please plan to merge PRs(including the > major > > > > > >> features) > > > > > >> > > > targeted > > > > > >> > > > > >> for > > > > > >> > > > > >> > > > > > 3.6.0 by the end of Aug 20th UTC. Starting > from > > > > > >> August > > > > > >> > 21st, > > > > > >> > > > > >> any pull > > > > > >> > > > > >> > > > > > requests intended for the 3.6.0 release > must > > > > include > > > > > >> the > > > > > >> > > > changes > > > > > >> > > > > >> > > > > > merged into the 3.6 branch as mentioned in > the > > > > > >> release > > > > > >> > plan. > > > > > >> > > > > >> > > > > > > > > > > >> > > > > >> > > > > > Thanks, > > > > > >> > > > > >> > > > > > Satish. > > > > > >> > > > > >> > > > > > > > > > > >> > > > > >> > > > > > On Fri, 4 Aug 2023 at 18:39, Chris Egerton > > > > > >> > > > > >> <chr...@aiven.io.invalid> > > > > > >> > > > > >> > > > > wrote: > > > > > >> > > > > >> > > > > > > > > > > > >> > > > > >> > > > > > > Thanks for adding KIP-949, Satish! > > > > > >> > > > > >> > > > > > > > > > > > >> > > > > >> > > > > > > On Fri, Aug 4, 2023 at 7:06 AM Satish > > > Duggana < > > > > > >> > > > > >> > > > > satish.dugg...@gmail.com> > > > > > >> > > > > >> > > > > > > wrote: > > > > > >> > > > > >> > > > > > > > > > > > >> > > > > >> > > > > > > > Hi, > > > > > >> > > > > >> > > > > > > > Myself and Divij discussed and added > the > > > > wiki for > > > > > >> > Kafka > > > > > >> > > > > >> > > TieredStorage > > > > > >> > > > > >> > > > > > > > Early Access Release[1]. If you have > any > > > > > >> comments or > > > > > >> > > > > >> feedback, > > > > > >> > > > > >> > > please > > > > > >> > > > > >> > > > > > > > feel free to share them. > > > > > >> > > > > >> > > > > > > > > > > > > >> > > > > >> > > > > > > > 1. > > > > > >> > > > > >> > > > > > > > > > > > > >> > > > > >> > > > > > > > > > >> > > > > >> > > > > > > > >> > > > > >> > > > > > >> > > > > > > > > >> > > > > > > >> > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Tiered+Storage+Early+Access+Release+Notes > > > > > >> > > > > >> > > > > > > > > > > > > >> > > > > >> > > > > > > > Thanks, > > > > > >> > > > > >> > > > > > > > Satish. > > > > > >> > > > > >> > > > > > > > > > > > > >> > > > > >> > > > > > > > On Fri, 4 Aug 2023 at 08:40, Satish > > > Duggana < > > > > > >> > > > > >> > > > > satish.dugg...@gmail.com> > > > > > >> > > > > >> > > > > > > > wrote: > > > > > >> > > > > >> > > > > > > > > > > > > > >> > > > > >> > > > > > > > > Hi Chris, > > > > > >> > > > > >> > > > > > > > > Thanks for the update. This looks to > be a > > > > minor > > > > > >> > change > > > > > >> > > > > >> and is > > > > > >> > > > > >> > > also > > > > > >> > > > > >> > > > > > > > > useful for backward compatibility. I > > > added > > > > it > > > > > >> to > > > > > >> > the > > > > > >> > > > > >> release > > > > > >> > > > > >> > > plan > > > > > >> > > > > >> > > > > as > > > > > >> > > > > >> > > > > > > > > an exceptional case. > > > > > >> > > > > >> > > > > > > > > > > > > > >> > > > > >> > > > > > > > > ~Satish. > > > > > >> > > > > >> > > > > > > > > > > > > > >> > > > > >> > > > > > > > > On Thu, 3 Aug 2023 at 21:34, Chris > > > Egerton > > > > > >> > > > > >> > > <chr...@aiven.io.invalid > > > > > >> > > > > >> > > > > > > > > > > >> > > > > >> > > > > > > > wrote: > > > > > >> > > > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > Hi Satish, > > > > > >> > > > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > Would it be possible to include > > > KIP-949 ( > > > > > >> > > > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > >> > > > > >> > > > > > > > > > >> > > > > >> > > > > > > > >> > > > > >> > > > > > >> > > > > > > > > >> > > > > > > >> > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-949%3A+Add+flag+to+enable+the+usage+of+topic+separator+in+MM2+DefaultReplicationPolicy > > > > > >> > > > > >> > > > > > > > ) > > > > > >> > > > > >> > > > > > > > > > in the 3.6.0 release? It passed > voting > > > > > >> > yesterday, > > > > > >> > > > and > > > > > >> > > > > >> is a > > > > > >> > > > > >> > > very > > > > > >> > > > > >> > > > > small, > > > > > >> > > > > >> > > > > > > > > > low-risk change that we'd like to > put > > > > out as > > > > > >> > soon as > > > > > >> > > > > >> > > possible in > > > > > >> > > > > >> > > > > order > > > > > >> > > > > >> > > > > > > > to > > > > > >> > > > > >> > > > > > > > > > patch an accidental break in > backwards > > > > > >> > compatibility > > > > > >> > > > > >> caused > > > > > >> > > > > >> > > a few > > > > > >> > > > > >> > > > > > > > versions > > > > > >> > > > > >> > > > > > > > > > ago. > > > > > >> > > > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > Best, > > > > > >> > > > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > Chris > > > > > >> > > > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > On Fri, Jul 28, 2023 at 2:35 AM > Satish > > > > > >> Duggana < > > > > > >> > > > > >> > > > > > > > satish.dugg...@gmail.com> > > > > > >> > > > > >> > > > > > > > > > wrote: > > > > > >> > > > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > Hi All, > > > > > >> > > > > >> > > > > > > > > > > Whoever has KIP entries in the > 3.6.0 > > > > > >> release > > > > > >> > plan. > > > > > >> > > > > >> Please > > > > > >> > > > > >> > > > > update it > > > > > >> > > > > >> > > > > > > > > > > with the latest status by > > > tomorrow(end > > > > of > > > > > >> the > > > > > >> > day > > > > > >> > > > > >> 29th Jul > > > > > >> > > > > >> > > UTC > > > > > >> > > > > >> > > > > ). > > > > > >> > > > > >> > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > Thanks > > > > > >> > > > > >> > > > > > > > > > > Satish. > > > > > >> > > > > >> > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > On Fri, 28 Jul 2023 at 12:01, > Satish > > > > > >> Duggana < > > > > > >> > > > > >> > > > > > > > satish.dugg...@gmail.com> > > > > > >> > > > > >> > > > > > > > > > > wrote: > > > > > >> > > > > >> > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > Thanks Ismael and Divij for the > > > > > >> suggestions. > > > > > >> > > > > >> > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > One way was to follow the > earlier > > > > > >> guidelines > > > > > >> > > > that > > > > > >> > > > > >> we set > > > > > >> > > > > >> > > for > > > > > >> > > > > >> > > > > any > > > > > >> > > > > >> > > > > > > > early > > > > > >> > > > > >> > > > > > > > > > > > access release. It looks Ismael > > > > already > > > > > >> > > > mentioned > > > > > >> > > > > >> the > > > > > >> > > > > >> > > > > example of > > > > > >> > > > > >> > > > > > > > > > > > KRaft. > > > > > >> > > > > >> > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > KIP-405 mentions > upgrade/downgrade > > > > and > > > > > >> > > > limitations > > > > > >> > > > > >> > > sections. > > > > > >> > > > > >> > > > > We can > > > > > >> > > > > >> > > > > > > > > > > > clarify that in the release > notes > > > for > > > > > >> users > > > > > >> > on > > > > > >> > > > how > > > > > >> > > > > >> this > > > > > >> > > > > >> > > > > feature > > > > > >> > > > > >> > > > > > > > can be > > > > > >> > > > > >> > > > > > > > > > > > used for early access. > > > > > >> > > > > >> > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > Divij, We do not want users to > > > enable > > > > > >> this > > > > > >> > > > feature > > > > > >> > > > > >> on > > > > > >> > > > > >> > > > > production > > > > > >> > > > > >> > > > > > > > > > > > environments in early access > > > release. > > > > > >> Let us > > > > > >> > > > work > > > > > >> > > > > >> > > together > > > > > >> > > > > >> > > > > on the > > > > > >> > > > > >> > > > > > > > > > > > followups Ismael suggested. > > > > > >> > > > > >> > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > ~Satish. > > > > > >> > > > > >> > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > On Fri, 28 Jul 2023 at 02:24, > Divij > > > > > >> Vaidya < > > > > > >> > > > > >> > > > > > > > divijvaidy...@gmail.com> > > > > > >> > > > > >> > > > > > > > > > > wrote: > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > Those are great suggestions, > > > thank > > > > > >> you. We > > > > > >> > > > will > > > > > >> > > > > >> > > continue > > > > > >> > > > > >> > > > > this > > > > > >> > > > > >> > > > > > > > > > > discussion > > > > > >> > > > > >> > > > > > > > > > > > > forward in a separate KIP for > > > > release > > > > > >> > plan for > > > > > >> > > > > >> Tiered > > > > > >> > > > > >> > > > > Storage. > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > On Thu 27. Jul 2023 at 21:46, > > > > Ismael > > > > > >> Juma > > > > > >> > < > > > > > >> > > > > >> > > > > m...@ismaeljuma.com> > > > > > >> > > > > >> > > > > > > > wrote: > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > Hi Divij, > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > I think the points you > bring up > > > > for > > > > > >> > > > discussion > > > > > >> > > > > >> are > > > > > >> > > > > >> > > all > > > > > >> > > > > >> > > > > good. > > > > > >> > > > > >> > > > > > > > My main > > > > > >> > > > > >> > > > > > > > > > > > > > feedback is that they > should be > > > > > >> > discussed > > > > > >> > > > in the > > > > > >> > > > > >> > > context > > > > > >> > > > > >> > > > > of > > > > > >> > > > > >> > > > > > > > KIPs vs > > > > > >> > > > > >> > > > > > > > > > > the > > > > > >> > > > > >> > > > > > > > > > > > > > release template. That's > why we > > > > have > > > > > >> a > > > > > >> > > > backwards > > > > > >> > > > > >> > > > > compatibility > > > > > >> > > > > >> > > > > > > > > > > section for > > > > > >> > > > > >> > > > > > > > > > > > > > every KIP, it's precisely > to > > > > ensure > > > > > >> we > > > > > >> > think > > > > > >> > > > > >> > > carefully > > > > > >> > > > > >> > > > > about > > > > > >> > > > > >> > > > > > > > some of > > > > > >> > > > > >> > > > > > > > > > > the > > > > > >> > > > > >> > > > > > > > > > > > > > points you're bringing up. > When > > > > it > > > > > >> > comes to > > > > > >> > > > > >> defining > > > > > >> > > > > >> > > the > > > > > >> > > > > >> > > > > > > > meaning of > > > > > >> > > > > >> > > > > > > > > > > early > > > > > >> > > > > >> > > > > > > > > > > > > > access, we have two > options: > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > 1. Have a KIP specifically > for > > > > tiered > > > > > >> > > > storage. > > > > > >> > > > > >> > > > > > > > > > > > > > 2. Have a KIP to define > general > > > > > >> > guidelines > > > > > >> > > > for > > > > > >> > > > > >> what > > > > > >> > > > > >> > > early > > > > > >> > > > > >> > > > > > > > access > > > > > >> > > > > >> > > > > > > > > > > means. > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > Does this make sense? > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > Ismael > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > On Thu, Jul 27, 2023 at > 6:38 PM > > > > Divij > > > > > >> > > > Vaidya < > > > > > >> > > > > >> > > > > > > > > > > divijvaidy...@gmail.com> > > > > > >> > > > > >> > > > > > > > > > > > > > wrote: > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > Thank you for the > response, > > > > Ismael. > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > 1. Specifically in > context of > > > > 3.6, > > > > > >> I > > > > > >> > > > wanted > > > > > >> > > > > >> this > > > > > >> > > > > >> > > > > > > > compatibility > > > > > >> > > > > >> > > > > > > > > > > > > > > guarantee point to > encourage > > > a > > > > > >> > discussion > > > > > >> > > > on > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > >> > > > > >> > > > > > > > > > >> > > > > >> > > > > > > > >> > > > > >> > > > > > >> > > > > > > > > >> > > > > > > >> > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-952%3A+Regenerate+segment-aligned+producer+snapshots+when+upgrading+to+a+Kafka+version+supporting+Tiered+Storage > > > > > >> > > > > >> > > > > > > > > > > > > > > . > > > > > >> > > > > >> > > > > > > > > > > > > > > Due to lack of producer > > > > snapshots > > > > > >> in > > > > > >> > <2.8 > > > > > >> > > > > >> > > versions, a > > > > > >> > > > > >> > > > > > > > customer may > > > > > >> > > > > >> > > > > > > > > > > not > > > > > >> > > > > >> > > > > > > > > > > > > > > be able to upgrade to > 3.6 and > > > > use > > > > > >> TS > > > > > >> > on a > > > > > >> > > > > >> topic > > > > > >> > > > > >> > > which > > > > > >> > > > > >> > > > > was > > > > > >> > > > > >> > > > > > > > created > > > > > >> > > > > >> > > > > > > > > > > when > > > > > >> > > > > >> > > > > > > > > > > > > > > the cluster was on <2.8 > > > version > > > > > >> (see > > > > > >> > > > > >> motivation for > > > > > >> > > > > >> > > > > > > > details). We > > > > > >> > > > > >> > > > > > > > > > > can > > > > > >> > > > > >> > > > > > > > > > > > > > > discuss and agree that it > > > does > > > > not > > > > > >> > break > > > > > >> > > > > >> > > compatibility, > > > > > >> > > > > >> > > > > > > > which is > > > > > >> > > > > >> > > > > > > > > > > fine. > > > > > >> > > > > >> > > > > > > > > > > > > > > But I want to ensure > that we > > > > have a > > > > > >> > > > discussion > > > > > >> > > > > >> > > soon on > > > > > >> > > > > >> > > > > this > > > > > >> > > > > >> > > > > > > > to > > > > > >> > > > > >> > > > > > > > > > > reach a > > > > > >> > > > > >> > > > > > > > > > > > > > > conclusion. > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > 2. I will start a KIP on > this > > > > for > > > > > >> > further > > > > > >> > > > > >> > > discussion. > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > 3. In the context of 3.6, > > > this > > > > > >> would > > > > > >> > mean > > > > > >> > > > that > > > > > >> > > > > >> > > there > > > > > >> > > > > >> > > > > should > > > > > >> > > > > >> > > > > > > > be > > > > > >> > > > > >> > > > > > > > > > > > > > > no-regression, if a user > does > > > > "not" > > > > > >> > > > turn-on > > > > > >> > > > > >> remote > > > > > >> > > > > >> > > > > storage > > > > > >> > > > > >> > > > > > > > (early > > > > > >> > > > > >> > > > > > > > > > > > > > > access feature) at a > cluster > > > > > >> level. We > > > > > >> > > > have > > > > > >> > > > > >> some > > > > > >> > > > > >> > > known > > > > > >> > > > > >> > > > > cases > > > > > >> > > > > >> > > > > > > > (such > > > > > >> > > > > >> > > > > > > > > > > as > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > >> https://issues.apache.org/jira/browse/KAFKA-15189) > > > > > >> > > > > >> > > > > which > > > > > >> > > > > >> > > > > > > > violate > > > > > >> > > > > >> > > > > > > > > > > this > > > > > >> > > > > >> > > > > > > > > > > > > > > compatibility > requirement. > > > > Having > > > > > >> this > > > > > >> > > > > >> guarantee > > > > > >> > > > > >> > > > > mentioned > > > > > >> > > > > >> > > > > > > > in the > > > > > >> > > > > >> > > > > > > > > > > > > > > release plan will ensure > that > > > > we > > > > > >> are > > > > > >> > all > > > > > >> > > > in > > > > > >> > > > > >> > > agreement > > > > > >> > > > > >> > > > > with > > > > > >> > > > > >> > > > > > > > which > > > > > >> > > > > >> > > > > > > > > > > cases > > > > > >> > > > > >> > > > > > > > > > > > > > > are truly blockers and > which > > > > > >> aren't. > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > 4. Fair, instead of a > general > > > > goal, > > > > > >> > let me > > > > > >> > > > > >> put it > > > > > >> > > > > >> > > > > > > > specifically in > > > > > >> > > > > >> > > > > > > > > > > the > > > > > >> > > > > >> > > > > > > > > > > > > > > context of 3.6. Let me > know > > > if > > > > > >> this is > > > > > >> > > > not the > > > > > >> > > > > >> > > right > > > > > >> > > > > >> > > > > forum > > > > > >> > > > > >> > > > > > > > for this > > > > > >> > > > > >> > > > > > > > > > > > > > > discussion. > > > > > >> > > > > >> > > > > > > > > > > > > > > Once a user "turns on" > tiered > > > > > >> storage > > > > > >> > > > (TS) at > > > > > >> > > > > >> a > > > > > >> > > > > >> > > cluster > > > > > >> > > > > >> > > > > > > > level, I am > > > > > >> > > > > >> > > > > > > > > > > > > > > proposing that they > should > > > > have the > > > > > >> > > > ability to > > > > > >> > > > > >> > > turn it > > > > > >> > > > > >> > > > > off > > > > > >> > > > > >> > > > > > > > as well > > > > > >> > > > > >> > > > > > > > > > > at > > > > > >> > > > > >> > > > > > > > > > > > > > > a cluster level. Since > this > > > is > > > > a > > > > > >> topic > > > > > >> > > > level > > > > > >> > > > > >> > > feature, > > > > > >> > > > > >> > > > > folks > > > > > >> > > > > >> > > > > > > > may not > > > > > >> > > > > >> > > > > > > > > > > > > > > spin up a separate > cluster to > > > > try > > > > > >> this > > > > > >> > > > > >> feature, > > > > > >> > > > > >> > > hence, > > > > > >> > > > > >> > > > > we > > > > > >> > > > > >> > > > > > > > need to > > > > > >> > > > > >> > > > > > > > > > > > > > > ensure that we provide > them > > > > with > > > > > >> the > > > > > >> > > > ability > > > > > >> > > > > >> to try > > > > > >> > > > > >> > > > > tiered > > > > > >> > > > > >> > > > > > > > storage > > > > > >> > > > > >> > > > > > > > > > > for > > > > > >> > > > > >> > > > > > > > > > > > > > > a topic which could be > > > deleted > > > > and > > > > > >> > > > featured > > > > > >> > > > > >> > > > > turned-off, so > > > > > >> > > > > >> > > > > > > > that > > > > > >> > > > > >> > > > > > > > > > > rest > > > > > >> > > > > >> > > > > > > > > > > > > > > of the production cases > are > > > not > > > > > >> > impacted. > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > 5. Agree on not making > public > > > > > >> > interface > > > > > >> > > > > >> change as a > > > > > >> > > > > >> > > > > > > > requirement > > > > > >> > > > > >> > > > > > > > > > > but we > > > > > >> > > > > >> > > > > > > > > > > > > > > should define what "early > > > > access" > > > > > >> > means in > > > > > >> > > > > >> that > > > > > >> > > > > >> > > case. > > > > > >> > > > > >> > > > > Users > > > > > >> > > > > >> > > > > > > > may > > > > > >> > > > > >> > > > > > > > > > > not be > > > > > >> > > > > >> > > > > > > > > > > > > > > aware that "early access" > > > > public > > > > > >> APIs > > > > > >> > may > > > > > >> > > > > >> change > > > > > >> > > > > >> > > > > (unless I am > > > > > >> > > > > >> > > > > > > > > > > missing > > > > > >> > > > > >> > > > > > > > > > > > > > > some documentation > somewhere > > > > > >> > completely, > > > > > >> > > > in > > > > > >> > > > > >> which > > > > > >> > > > > >> > > case > > > > > >> > > > > >> > > > > I > > > > > >> > > > > >> > > > > > > > apologize > > > > > >> > > > > >> > > > > > > > > > > for > > > > > >> > > > > >> > > > > > > > > > > > > > > bringing this naive > point). > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > -- > > > > > >> > > > > >> > > > > > > > > > > > > > > Divij Vaidya > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > On Thu, Jul 27, 2023 at > > > 2:27 PM > > > > > >> Ismael > > > > > >> > > > Juma < > > > > > >> > > > > >> > > > > > > > m...@ismaeljuma.com> > > > > > >> > > > > >> > > > > > > > > > > wrote: > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > Hi Divij, > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > Some of these are > launch > > > > > >> checklist > > > > > >> > items > > > > > >> > > > > >> (not > > > > > >> > > > > >> > > really > > > > > >> > > > > >> > > > > > > > goals) and > > > > > >> > > > > >> > > > > > > > > > > some > > > > > >> > > > > >> > > > > > > > > > > > > > are > > > > > >> > > > > >> > > > > > > > > > > > > > > > compatibility > guarantees. > > > > More > > > > > >> > below. > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > On Thu, Jul 27, 2023, > 12:10 > > > > PM > > > > > >> Divij > > > > > >> > > > Vaidya > > > > > >> > > > > >> < > > > > > >> > > > > >> > > > > > > > > > > divijvaidy...@gmail.com> > > > > > >> > > > > >> > > > > > > > > > > > > > > wrote: > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > Hey Satish > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > Could we consider > adding > > > > > >> "launch > > > > > >> > > > goals" > > > > > >> > > > > >> in the > > > > > >> > > > > >> > > > > release > > > > > >> > > > > >> > > > > > > > plan. > > > > > >> > > > > >> > > > > > > > > > > While > > > > > >> > > > > >> > > > > > > > > > > > > > > > > some of these may be > > > > implicit, > > > > > >> it > > > > > >> > > > would be > > > > > >> > > > > >> > > nice to > > > > > >> > > > > >> > > > > list > > > > > >> > > > > >> > > > > > > > them > > > > > >> > > > > >> > > > > > > > > > > down in > > > > > >> > > > > >> > > > > > > > > > > > > > > > > the release plan. For > > > this > > > > > >> > release, > > > > > >> > > > our > > > > > >> > > > > >> launch > > > > > >> > > > > >> > > > > > > > requirements > > > > > >> > > > > >> > > > > > > > > > > would be: > > > > > >> > > > > >> > > > > > > > > > > > > > > > > 1. Users should be > able > > > to > > > > > >> upgrade > > > > > >> > > > from > > > > > >> > > > > >> any > > > > > >> > > > > >> > > prior > > > > > >> > > > > >> > > > > Kafka > > > > > >> > > > > >> > > > > > > > > > > version to > > > > > >> > > > > >> > > > > > > > > > > > > > this > > > > > >> > > > > >> > > > > > > > > > > > > > > > > version. > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > This is part of the > > > > compatibility > > > > > >> > > > > >> guarantees. The > > > > > >> > > > > >> > > > > upgrade > > > > > >> > > > > >> > > > > > > > notes > > > > > >> > > > > >> > > > > > > > > > > mention > > > > > >> > > > > >> > > > > > > > > > > > > > > > this already. If there > is a > > > > > >> change > > > > > >> > in a > > > > > >> > > > > >> given > > > > > >> > > > > >> > > > > release, it > > > > > >> > > > > >> > > > > > > > should > > > > > >> > > > > >> > > > > > > > > > > > > > > definitely > > > > > >> > > > > >> > > > > > > > > > > > > > > > be highlighted. > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > 2. On release, this > version > > > > (or > > > > > >> it's > > > > > >> > > > > >> > > dependencies) > > > > > >> > > > > >> > > > > would > > > > > >> > > > > >> > > > > > > > not > > > > > >> > > > > >> > > > > > > > > > > have any > > > > > >> > > > > >> > > > > > > > > > > > > > > > > known MEDIUM/HIGH > CVE. > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > This is a new policy > and > > > the > > > > > >> details > > > > > >> > > > should > > > > > >> > > > > >> be > > > > > >> > > > > >> > > > > discussed. > > > > > >> > > > > >> > > > > > > > In > > > > > >> > > > > >> > > > > > > > > > > > > > particular, > > > > > >> > > > > >> > > > > > > > > > > > > > > > the threshold (medium > or > > > > high). > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > 3. Presence of any > "early > > > > > >> > access"/"beta" > > > > > >> > > > > >> feature > > > > > >> > > > > >> > > > > should not > > > > > >> > > > > >> > > > > > > > > > > impact > > > > > >> > > > > >> > > > > > > > > > > > > > > > > other production > features > > > > when > > > > > >> it > > > > > >> > is > > > > > >> > > > not > > > > > >> > > > > >> > > enabled. > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > This is a general > guideline > > > > for > > > > > >> > early > > > > > >> > > > access > > > > > >> > > > > >> > > > > features and > > > > > >> > > > > >> > > > > > > > not > > > > > >> > > > > >> > > > > > > > > > > specific > > > > > >> > > > > >> > > > > > > > > > > > > > to > > > > > >> > > > > >> > > > > > > > > > > > > > > > this release. It would > be > > > > good to > > > > > >> > have a > > > > > >> > > > > >> page > > > > > >> > > > > >> > > that > > > > > >> > > > > >> > > > > talks > > > > > >> > > > > >> > > > > > > > about > > > > > >> > > > > >> > > > > > > > > > > these > > > > > >> > > > > >> > > > > > > > > > > > > > > things. > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > 4. Once enabled, users > > > should > > > > > >> have > > > > > >> > an > > > > > >> > > > > >> option to > > > > > >> > > > > >> > > > > disable any > > > > > >> > > > > >> > > > > > > > > > > "early > > > > > >> > > > > >> > > > > > > > > > > > > > > > > access"/"beta" > feature > > > and > > > > > >> resume > > > > > >> > > > normal > > > > > >> > > > > >> > > production > > > > > >> > > > > >> > > > > > > > features, > > > > > >> > > > > >> > > > > > > > > > > i.e. > > > > > >> > > > > >> > > > > > > > > > > > > > > > > impact of beta > features > > > > should > > > > > >> be > > > > > >> > > > > >> reversible. > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > This needs discussion > and I > > > > don't > > > > > >> > think > > > > > >> > > > it's > > > > > >> > > > > >> > > > > reasonable as > > > > > >> > > > > >> > > > > > > > a > > > > > >> > > > > >> > > > > > > > > > > general > > > > > >> > > > > >> > > > > > > > > > > > > > > rule. > > > > > >> > > > > >> > > > > > > > > > > > > > > > For example, Kraft > early > > > > access > > > > > >> > wasn't > > > > > >> > > > > >> reversible > > > > > >> > > > > >> > > > > and it > > > > > >> > > > > >> > > > > > > > was not > > > > > >> > > > > >> > > > > > > > > > > > > > feasible > > > > > >> > > > > >> > > > > > > > > > > > > > > > for it to be. > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > 5. KIP-405 will be > > > available > > > > in > > > > > >> > "early > > > > > >> > > > > >> > > access"/"beta" > > > > > >> > > > > >> > > > > > > > mode. Early > > > > > >> > > > > >> > > > > > > > > > > > > > > > > access/beta means > that > > > the > > > > > >> public > > > > > >> > > > facing > > > > > >> > > > > >> > > > > interfaces won't > > > > > >> > > > > >> > > > > > > > > > > change in > > > > > >> > > > > >> > > > > > > > > > > > > > > > > future but the > > > > implementation > > > > > >> is > > > > > >> > not > > > > > >> > > > > >> > > recommended > > > > > >> > > > > >> > > > > to be > > > > > >> > > > > >> > > > > > > > used in > > > > > >> > > > > >> > > > > > > > > > > > > > > > > production. > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > I don't think it's ok > to > > > make > > > > > >> this a > > > > > >> > > > > >> requirement. > > > > > >> > > > > >> > > > > Early > > > > > >> > > > > >> > > > > > > > access > > > > > >> > > > > >> > > > > > > > > > > is a way > > > > > >> > > > > >> > > > > > > > > > > > > > > to > > > > > >> > > > > >> > > > > > > > > > > > > > > > get early feedback and > all > > > > types > > > > > >> of > > > > > >> > > > changes > > > > > >> > > > > >> > > should > > > > > >> > > > > >> > > > > be on > > > > > >> > > > > >> > > > > > > > the > > > > > >> > > > > >> > > > > > > > > > > table. > > > > > >> > > > > >> > > > > > > > > > > > > > They > > > > > >> > > > > >> > > > > > > > > > > > > > > > would be discussed via > KIPs > > > > as > > > > > >> > usual. I > > > > > >> > > > > >> believe > > > > > >> > > > > >> > > > > there were > > > > > >> > > > > >> > > > > > > > some > > > > > >> > > > > >> > > > > > > > > > > > > > > > incompatible changes > for > > > > Kraft > > > > > >> > during > > > > > >> > > > the > > > > > >> > > > > >> early > > > > > >> > > > > >> > > > > access > > > > > >> > > > > >> > > > > > > > period > > > > > >> > > > > >> > > > > > > > > > > although > > > > > >> > > > > >> > > > > > > > > > > > > > > the > > > > > >> > > > > >> > > > > > > > > > > > > > > > team aimed to minimize > work > > > > > >> required > > > > > >> > > > during > > > > > >> > > > > >> > > > > upgrades. I > > > > > >> > > > > >> > > > > > > > have > > > > > >> > > > > >> > > > > > > > > > > mentioned > > > > > >> > > > > >> > > > > > > > > > > > > > > > Kraft a couple of times > > > since > > > > > >> it's a > > > > > >> > > > good > > > > > >> > > > > >> > > example of > > > > > >> > > > > >> > > > > a > > > > > >> > > > > >> > > > > > > > large > > > > > >> > > > > >> > > > > > > > > > > feature > > > > > >> > > > > >> > > > > > > > > > > > > > that > > > > > >> > > > > >> > > > > > > > > > > > > > > > went through this > process. > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > Ismael > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > -- > > > > > >> > > > > >> > > > > > > > > > > > > Divij Vaidya > > > > > >> > > > > >> > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > >> > > > > >> > > > > > > > > > >> > > > > >> > > > > > > > >> > > > > >> > > > > > >> > > > > > > > > > > >> > > > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > > > > -- > > David Arthur > > -- David Arthur
