Hi Satish, Wanted to let you know that KAFKA-12879 ( https://issues.apache.org/jira/browse/KAFKA-12879), a breaking change in Admin::listOffsets, has been reintroduced into the code base. Since we haven't yet published a release with this change (at least, not the more recent instance of it), I was hoping we could treat it as a blocker for 3.6.0. I'd also like to solicit the input of people familiar with the admin client to weigh in on the Jira ticket about whether we should continue to preserve the current behavior (if the consensus is that we should, I'm happy to file a fix).
Please let me know if you agree that this qualifies as a blocker. I plan on publishing a potential fix sometime this week. Cheers, Chris On Wed, Aug 30, 2023 at 9:19 AM Satish Duggana <satish.dugg...@gmail.com> wrote: > Hi, > Please plan to continue merging pull requests associated with any > outstanding minor features and stabilization changes to 3.6 branch > before September 3rd. Kindly update the KIP's implementation status in > the 3.6.0 release notes. > > Thanks, > Satish. > > On Fri, 25 Aug 2023 at 21:37, Justine Olshan > <jols...@confluent.io.invalid> wrote: > > > > Hey Satish, > > Everything should be in 3.6, and I will update the release plan wiki. > > Thanks! > > > > On Fri, Aug 25, 2023 at 4:08 AM Satish Duggana <satish.dugg...@gmail.com > > > > wrote: > > > > > Hi Justine, > > > Adding KIP-890 part-1 to 3.6.0 seems reasonable to me. This part looks > > > to be addressing a critical issue of consumers getting stuck. Please > > > update the release plan wiki and merge all the required changes to 3.6 > > > branch. > > > > > > Thanks, > > > Satish. > > > > > > On Thu, 24 Aug 2023 at 22:19, Justine Olshan > > > <jols...@confluent.io.invalid> wrote: > > > > > > > > Hey Satish, > > > > Does it make sense to include KIP-890 part 1? It prevents hanging > > > > transactions for older clients. (An optimization and stronger EOS > > > > guarantees will be included in part 2) > > > > > > > > Thanks, > > > > Justine > > > > > > > > On Mon, Aug 21, 2023 at 3:29 AM Satish Duggana < > satish.dugg...@gmail.com > > > > > > > > wrote: > > > > > > > > > Hi, > > > > > 3.6 branch is created. Please make sure any PRs targeted for 3.6.0 > > > > > should be merged to 3.6 branch once those are merged to trunk. > > > > > > > > > > Thanks, > > > > > Satish. > > > > > > > > > > On Wed, 16 Aug 2023 at 15:58, Satish Duggana < > satish.dugg...@gmail.com > > > > > > > > > wrote: > > > > > > > > > > > > Hi, > > > > > > Please plan to merge PRs(including the major features) targeted > for > > > > > > 3.6.0 by the end of Aug 20th UTC. Starting from August 21st, any > pull > > > > > > requests intended for the 3.6.0 release must include the changes > > > > > > merged into the 3.6 branch as mentioned in the release plan. > > > > > > > > > > > > Thanks, > > > > > > Satish. > > > > > > > > > > > > On Fri, 4 Aug 2023 at 18:39, Chris Egerton > <chr...@aiven.io.invalid> > > > > > wrote: > > > > > > > > > > > > > > Thanks for adding KIP-949, Satish! > > > > > > > > > > > > > > On Fri, Aug 4, 2023 at 7:06 AM Satish Duggana < > > > > > satish.dugg...@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > Hi, > > > > > > > > Myself and Divij discussed and added the wiki for Kafka > > > TieredStorage > > > > > > > > Early Access Release[1]. If you have any comments or > feedback, > > > please > > > > > > > > feel free to share them. > > > > > > > > > > > > > > > > 1. > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Tiered+Storage+Early+Access+Release+Notes > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Satish. > > > > > > > > > > > > > > > > On Fri, 4 Aug 2023 at 08:40, Satish Duggana < > > > > > satish.dugg...@gmail.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > Hi Chris, > > > > > > > > > Thanks for the update. This looks to be a minor change and > is > > > also > > > > > > > > > useful for backward compatibility. I added it to the > release > > > plan > > > > > as > > > > > > > > > an exceptional case. > > > > > > > > > > > > > > > > > > ~Satish. > > > > > > > > > > > > > > > > > > On Thu, 3 Aug 2023 at 21:34, Chris Egerton > > > <chr...@aiven.io.invalid > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > Hi Satish, > > > > > > > > > > > > > > > > > > > > Would it be possible to include KIP-949 ( > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-949%3A+Add+flag+to+enable+the+usage+of+topic+separator+in+MM2+DefaultReplicationPolicy > > > > > > > > ) > > > > > > > > > > in the 3.6.0 release? It passed voting yesterday, and is > a > > > very > > > > > small, > > > > > > > > > > low-risk change that we'd like to put out as soon as > > > possible in > > > > > order > > > > > > > > to > > > > > > > > > > patch an accidental break in backwards compatibility > caused > > > a few > > > > > > > > versions > > > > > > > > > > ago. > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > > > > > > > > Chris > > > > > > > > > > > > > > > > > > > > On Fri, Jul 28, 2023 at 2:35 AM Satish Duggana < > > > > > > > > satish.dugg...@gmail.com> > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > Whoever has KIP entries in the 3.6.0 release plan. > Please > > > > > update it > > > > > > > > > > > with the latest status by tomorrow(end of the day 29th > Jul > > > UTC > > > > > ). > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > > > > Satish. > > > > > > > > > > > > > > > > > > > > > > On Fri, 28 Jul 2023 at 12:01, Satish Duggana < > > > > > > > > satish.dugg...@gmail.com> > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > Thanks Ismael and Divij for the suggestions. > > > > > > > > > > > > > > > > > > > > > > > > One way was to follow the earlier guidelines that we > set > > > for > > > > > any > > > > > > > > early > > > > > > > > > > > > access release. It looks Ismael already mentioned the > > > > > example of > > > > > > > > > > > > KRaft. > > > > > > > > > > > > > > > > > > > > > > > > KIP-405 mentions upgrade/downgrade and limitations > > > sections. > > > > > We can > > > > > > > > > > > > clarify that in the release notes for users on how > this > > > > > feature > > > > > > > > can be > > > > > > > > > > > > used for early access. > > > > > > > > > > > > > > > > > > > > > > > > Divij, We do not want users to enable this feature on > > > > > production > > > > > > > > > > > > environments in early access release. Let us work > > > together > > > > > on the > > > > > > > > > > > > followups Ismael suggested. > > > > > > > > > > > > > > > > > > > > > > > > ~Satish. > > > > > > > > > > > > > > > > > > > > > > > > On Fri, 28 Jul 2023 at 02:24, Divij Vaidya < > > > > > > > > divijvaidy...@gmail.com> > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > Those are great suggestions, thank you. We will > > > continue > > > > > this > > > > > > > > > > > discussion > > > > > > > > > > > > > forward in a separate KIP for release plan for > Tiered > > > > > Storage. > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu 27. Jul 2023 at 21:46, Ismael Juma < > > > > > m...@ismaeljuma.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Divij, > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think the points you bring up for discussion > are > > > all > > > > > good. > > > > > > > > My main > > > > > > > > > > > > > > feedback is that they should be discussed in the > > > context > > > > > of > > > > > > > > KIPs vs > > > > > > > > > > > the > > > > > > > > > > > > > > release template. That's why we have a backwards > > > > > compatibility > > > > > > > > > > > section for > > > > > > > > > > > > > > every KIP, it's precisely to ensure we think > > > carefully > > > > > about > > > > > > > > some of > > > > > > > > > > > the > > > > > > > > > > > > > > points you're bringing up. When it comes to > defining > > > the > > > > > > > > meaning of > > > > > > > > > > > early > > > > > > > > > > > > > > access, we have two options: > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. Have a KIP specifically for tiered storage. > > > > > > > > > > > > > > 2. Have a KIP to define general guidelines for > what > > > early > > > > > > > > access > > > > > > > > > > > means. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Does this make sense? > > > > > > > > > > > > > > > > > > > > > > > > > > > > Ismael > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Jul 27, 2023 at 6:38 PM Divij Vaidya < > > > > > > > > > > > divijvaidy...@gmail.com> > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thank you for the response, Ismael. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. Specifically in context of 3.6, I wanted > this > > > > > > > > compatibility > > > > > > > > > > > > > > > guarantee point to encourage a discussion on > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-952%3A+Regenerate+segment-aligned+producer+snapshots+when+upgrading+to+a+Kafka+version+supporting+Tiered+Storage > > > > > > > > > > > > > > > . > > > > > > > > > > > > > > > Due to lack of producer snapshots in <2.8 > > > versions, a > > > > > > > > customer may > > > > > > > > > > > not > > > > > > > > > > > > > > > be able to upgrade to 3.6 and use TS on a topic > > > which > > > > > was > > > > > > > > created > > > > > > > > > > > when > > > > > > > > > > > > > > > the cluster was on <2.8 version (see > motivation for > > > > > > > > details). We > > > > > > > > > > > can > > > > > > > > > > > > > > > discuss and agree that it does not break > > > compatibility, > > > > > > > > which is > > > > > > > > > > > fine. > > > > > > > > > > > > > > > But I want to ensure that we have a discussion > > > soon on > > > > > this > > > > > > > > to > > > > > > > > > > > reach a > > > > > > > > > > > > > > > conclusion. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2. I will start a KIP on this for further > > > discussion. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 3. In the context of 3.6, this would mean that > > > there > > > > > should > > > > > > > > be > > > > > > > > > > > > > > > no-regression, if a user does "not" turn-on > remote > > > > > storage > > > > > > > > (early > > > > > > > > > > > > > > > access feature) at a cluster level. We have > some > > > known > > > > > cases > > > > > > > > (such > > > > > > > > > > > as > > > > > > > > > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-15189) > > > > > which > > > > > > > > violate > > > > > > > > > > > this > > > > > > > > > > > > > > > compatibility requirement. Having this > guarantee > > > > > mentioned > > > > > > > > in the > > > > > > > > > > > > > > > release plan will ensure that we are all in > > > agreement > > > > > with > > > > > > > > which > > > > > > > > > > > cases > > > > > > > > > > > > > > > are truly blockers and which aren't. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 4. Fair, instead of a general goal, let me put > it > > > > > > > > specifically in > > > > > > > > > > > the > > > > > > > > > > > > > > > context of 3.6. Let me know if this is not the > > > right > > > > > forum > > > > > > > > for this > > > > > > > > > > > > > > > discussion. > > > > > > > > > > > > > > > Once a user "turns on" tiered storage (TS) at a > > > cluster > > > > > > > > level, I am > > > > > > > > > > > > > > > proposing that they should have the ability to > > > turn it > > > > > off > > > > > > > > as well > > > > > > > > > > > at > > > > > > > > > > > > > > > a cluster level. Since this is a topic level > > > feature, > > > > > folks > > > > > > > > may not > > > > > > > > > > > > > > > spin up a separate cluster to try this feature, > > > hence, > > > > > we > > > > > > > > need to > > > > > > > > > > > > > > > ensure that we provide them with the ability > to try > > > > > tiered > > > > > > > > storage > > > > > > > > > > > for > > > > > > > > > > > > > > > a topic which could be deleted and featured > > > > > turned-off, so > > > > > > > > that > > > > > > > > > > > rest > > > > > > > > > > > > > > > of the production cases are not impacted. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 5. Agree on not making public interface change > as a > > > > > > > > requirement > > > > > > > > > > > but we > > > > > > > > > > > > > > > should define what "early access" means in that > > > case. > > > > > Users > > > > > > > > may > > > > > > > > > > > not be > > > > > > > > > > > > > > > aware that "early access" public APIs may > change > > > > > (unless I am > > > > > > > > > > > missing > > > > > > > > > > > > > > > some documentation somewhere completely, in > which > > > case > > > > > I > > > > > > > > apologize > > > > > > > > > > > for > > > > > > > > > > > > > > > bringing this naive point). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > Divij Vaidya > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Jul 27, 2023 at 2:27 PM Ismael Juma < > > > > > > > > m...@ismaeljuma.com> > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Divij, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Some of these are launch checklist items (not > > > really > > > > > > > > goals) and > > > > > > > > > > > some > > > > > > > > > > > > > > are > > > > > > > > > > > > > > > > compatibility guarantees. More below. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Jul 27, 2023, 12:10 PM Divij Vaidya < > > > > > > > > > > > divijvaidy...@gmail.com> > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hey Satish > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Could we consider adding "launch goals" in > the > > > > > release > > > > > > > > plan. > > > > > > > > > > > While > > > > > > > > > > > > > > > > > some of these may be implicit, it would be > > > nice to > > > > > list > > > > > > > > them > > > > > > > > > > > down in > > > > > > > > > > > > > > > > > the release plan. For this release, our > launch > > > > > > > > requirements > > > > > > > > > > > would be: > > > > > > > > > > > > > > > > > 1. Users should be able to upgrade from any > > > prior > > > > > Kafka > > > > > > > > > > > version to > > > > > > > > > > > > > > this > > > > > > > > > > > > > > > > > version. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This is part of the compatibility > guarantees. The > > > > > upgrade > > > > > > > > notes > > > > > > > > > > > mention > > > > > > > > > > > > > > > > this already. If there is a change in a given > > > > > release, it > > > > > > > > should > > > > > > > > > > > > > > > definitely > > > > > > > > > > > > > > > > be highlighted. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2. On release, this version (or it's > > > dependencies) > > > > > would > > > > > > > > not > > > > > > > > > > > have any > > > > > > > > > > > > > > > > > known MEDIUM/HIGH CVE. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This is a new policy and the details should > be > > > > > discussed. > > > > > > > > In > > > > > > > > > > > > > > particular, > > > > > > > > > > > > > > > > the threshold (medium or high). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 3. Presence of any "early access"/"beta" > feature > > > > > should not > > > > > > > > > > > impact > > > > > > > > > > > > > > > > > other production features when it is not > > > enabled. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This is a general guideline for early access > > > > > features and > > > > > > > > not > > > > > > > > > > > specific > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > this release. It would be good to have a page > > > that > > > > > talks > > > > > > > > about > > > > > > > > > > > these > > > > > > > > > > > > > > > things. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 4. Once enabled, users should have an option > to > > > > > disable any > > > > > > > > > > > "early > > > > > > > > > > > > > > > > > access"/"beta" feature and resume normal > > > production > > > > > > > > features, > > > > > > > > > > > i.e. > > > > > > > > > > > > > > > > > impact of beta features should be > reversible. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This needs discussion and I don't think it's > > > > > reasonable as > > > > > > > > a > > > > > > > > > > > general > > > > > > > > > > > > > > > rule. > > > > > > > > > > > > > > > > For example, Kraft early access wasn't > reversible > > > > > and it > > > > > > > > was not > > > > > > > > > > > > > > feasible > > > > > > > > > > > > > > > > for it to be. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 5. KIP-405 will be available in "early > > > access"/"beta" > > > > > > > > mode. Early > > > > > > > > > > > > > > > > > access/beta means that the public facing > > > > > interfaces won't > > > > > > > > > > > change in > > > > > > > > > > > > > > > > > future but the implementation is not > > > recommended > > > > > to be > > > > > > > > used in > > > > > > > > > > > > > > > > > production. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I don't think it's ok to make this a > requirement. > > > > > Early > > > > > > > > access > > > > > > > > > > > is a way > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > get early feedback and all types of changes > > > should > > > > > be on > > > > > > > > the > > > > > > > > > > > table. > > > > > > > > > > > > > > They > > > > > > > > > > > > > > > > would be discussed via KIPs as usual. I > believe > > > > > there were > > > > > > > > some > > > > > > > > > > > > > > > > incompatible changes for Kraft during the > early > > > > > access > > > > > > > > period > > > > > > > > > > > although > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > team aimed to minimize work required during > > > > > upgrades. I > > > > > > > > have > > > > > > > > > > > mentioned > > > > > > > > > > > > > > > > Kraft a couple of times since it's a good > > > example of > > > > > a > > > > > > > > large > > > > > > > > > > > feature > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > went through this process. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Ismael > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > Divij Vaidya > > > > > > > > > > > > > > > > > > > > > > > > > > > >
