Hi, Please plan to merge PRs(including the major features) targeted for 3.6.0 by the end of Aug 20th UTC. Starting from August 21st, any pull requests intended for the 3.6.0 release must include the changes merged into the 3.6 branch as mentioned in the release plan.
Thanks, Satish. On Fri, 4 Aug 2023 at 18:39, Chris Egerton <chr...@aiven.io.invalid> wrote: > > Thanks for adding KIP-949, Satish! > > On Fri, Aug 4, 2023 at 7:06 AM Satish Duggana <satish.dugg...@gmail.com> > wrote: > > > Hi, > > Myself and Divij discussed and added the wiki for Kafka TieredStorage > > Early Access Release[1]. If you have any comments or feedback, please > > feel free to share them. > > > > 1. > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Tiered+Storage+Early+Access+Release+Notes > > > > Thanks, > > Satish. > > > > On Fri, 4 Aug 2023 at 08:40, Satish Duggana <satish.dugg...@gmail.com> > > wrote: > > > > > > Hi Chris, > > > Thanks for the update. This looks to be a minor change and is also > > > useful for backward compatibility. I added it to the release plan as > > > an exceptional case. > > > > > > ~Satish. > > > > > > On Thu, 3 Aug 2023 at 21:34, Chris Egerton <chr...@aiven.io.invalid> > > wrote: > > > > > > > > Hi Satish, > > > > > > > > Would it be possible to include KIP-949 ( > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-949%3A+Add+flag+to+enable+the+usage+of+topic+separator+in+MM2+DefaultReplicationPolicy > > ) > > > > in the 3.6.0 release? It passed voting yesterday, and is a very small, > > > > low-risk change that we'd like to put out as soon as possible in order > > to > > > > patch an accidental break in backwards compatibility caused a few > > versions > > > > ago. > > > > > > > > Best, > > > > > > > > Chris > > > > > > > > On Fri, Jul 28, 2023 at 2:35 AM Satish Duggana < > > satish.dugg...@gmail.com> > > > > wrote: > > > > > > > > > Hi All, > > > > > Whoever has KIP entries in the 3.6.0 release plan. Please update it > > > > > with the latest status by tomorrow(end of the day 29th Jul UTC ). > > > > > > > > > > Thanks > > > > > Satish. > > > > > > > > > > On Fri, 28 Jul 2023 at 12:01, Satish Duggana < > > satish.dugg...@gmail.com> > > > > > wrote: > > > > > > > > > > > > Thanks Ismael and Divij for the suggestions. > > > > > > > > > > > > One way was to follow the earlier guidelines that we set for any > > early > > > > > > access release. It looks Ismael already mentioned the example of > > > > > > KRaft. > > > > > > > > > > > > KIP-405 mentions upgrade/downgrade and limitations sections. We can > > > > > > clarify that in the release notes for users on how this feature > > can be > > > > > > used for early access. > > > > > > > > > > > > Divij, We do not want users to enable this feature on production > > > > > > environments in early access release. Let us work together on the > > > > > > followups Ismael suggested. > > > > > > > > > > > > ~Satish. > > > > > > > > > > > > On Fri, 28 Jul 2023 at 02:24, Divij Vaidya < > > divijvaidy...@gmail.com> > > > > > wrote: > > > > > > > > > > > > > > Those are great suggestions, thank you. We will continue this > > > > > discussion > > > > > > > forward in a separate KIP for release plan for Tiered Storage. > > > > > > > > > > > > > > On Thu 27. Jul 2023 at 21:46, Ismael Juma <m...@ismaeljuma.com> > > wrote: > > > > > > > > > > > > > > > Hi Divij, > > > > > > > > > > > > > > > > I think the points you bring up for discussion are all good. > > My main > > > > > > > > feedback is that they should be discussed in the context of > > KIPs vs > > > > > the > > > > > > > > release template. That's why we have a backwards compatibility > > > > > section for > > > > > > > > every KIP, it's precisely to ensure we think carefully about > > some of > > > > > the > > > > > > > > points you're bringing up. When it comes to defining the > > meaning of > > > > > early > > > > > > > > access, we have two options: > > > > > > > > > > > > > > > > 1. Have a KIP specifically for tiered storage. > > > > > > > > 2. Have a KIP to define general guidelines for what early > > access > > > > > means. > > > > > > > > > > > > > > > > Does this make sense? > > > > > > > > > > > > > > > > Ismael > > > > > > > > > > > > > > > > On Thu, Jul 27, 2023 at 6:38 PM Divij Vaidya < > > > > > divijvaidy...@gmail.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Thank you for the response, Ismael. > > > > > > > > > > > > > > > > > > 1. Specifically in context of 3.6, I wanted this > > compatibility > > > > > > > > > guarantee point to encourage a discussion on > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-952%3A+Regenerate+segment-aligned+producer+snapshots+when+upgrading+to+a+Kafka+version+supporting+Tiered+Storage > > > > > > > > > . > > > > > > > > > Due to lack of producer snapshots in <2.8 versions, a > > customer may > > > > > not > > > > > > > > > be able to upgrade to 3.6 and use TS on a topic which was > > created > > > > > when > > > > > > > > > the cluster was on <2.8 version (see motivation for > > details). We > > > > > can > > > > > > > > > discuss and agree that it does not break compatibility, > > which is > > > > > fine. > > > > > > > > > But I want to ensure that we have a discussion soon on this > > to > > > > > reach a > > > > > > > > > conclusion. > > > > > > > > > > > > > > > > > > 2. I will start a KIP on this for further discussion. > > > > > > > > > > > > > > > > > > 3. In the context of 3.6, this would mean that there should > > be > > > > > > > > > no-regression, if a user does "not" turn-on remote storage > > (early > > > > > > > > > access feature) at a cluster level. We have some known cases > > (such > > > > > as > > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-15189) which > > violate > > > > > this > > > > > > > > > compatibility requirement. Having this guarantee mentioned > > in the > > > > > > > > > release plan will ensure that we are all in agreement with > > which > > > > > cases > > > > > > > > > are truly blockers and which aren't. > > > > > > > > > > > > > > > > > > 4. Fair, instead of a general goal, let me put it > > specifically in > > > > > the > > > > > > > > > context of 3.6. Let me know if this is not the right forum > > for this > > > > > > > > > discussion. > > > > > > > > > Once a user "turns on" tiered storage (TS) at a cluster > > level, I am > > > > > > > > > proposing that they should have the ability to turn it off > > as well > > > > > at > > > > > > > > > a cluster level. Since this is a topic level feature, folks > > may not > > > > > > > > > spin up a separate cluster to try this feature, hence, we > > need to > > > > > > > > > ensure that we provide them with the ability to try tiered > > storage > > > > > for > > > > > > > > > a topic which could be deleted and featured turned-off, so > > that > > > > > rest > > > > > > > > > of the production cases are not impacted. > > > > > > > > > > > > > > > > > > 5. Agree on not making public interface change as a > > requirement > > > > > but we > > > > > > > > > should define what "early access" means in that case. Users > > may > > > > > not be > > > > > > > > > aware that "early access" public APIs may change (unless I am > > > > > missing > > > > > > > > > some documentation somewhere completely, in which case I > > apologize > > > > > for > > > > > > > > > bringing this naive point). > > > > > > > > > > > > > > > > > > -- > > > > > > > > > Divij Vaidya > > > > > > > > > > > > > > > > > > On Thu, Jul 27, 2023 at 2:27 PM Ismael Juma < > > m...@ismaeljuma.com> > > > > > wrote: > > > > > > > > > > > > > > > > > > > > Hi Divij, > > > > > > > > > > > > > > > > > > > > Some of these are launch checklist items (not really > > goals) and > > > > > some > > > > > > > > are > > > > > > > > > > compatibility guarantees. More below. > > > > > > > > > > > > > > > > > > > > On Thu, Jul 27, 2023, 12:10 PM Divij Vaidya < > > > > > divijvaidy...@gmail.com> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Hey Satish > > > > > > > > > > > > > > > > > > > > > > Could we consider adding "launch goals" in the release > > plan. > > > > > While > > > > > > > > > > > some of these may be implicit, it would be nice to list > > them > > > > > down in > > > > > > > > > > > the release plan. For this release, our launch > > requirements > > > > > would be: > > > > > > > > > > > 1. Users should be able to upgrade from any prior Kafka > > > > > version to > > > > > > > > this > > > > > > > > > > > version. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This is part of the compatibility guarantees. The upgrade > > notes > > > > > mention > > > > > > > > > > this already. If there is a change in a given release, it > > should > > > > > > > > > definitely > > > > > > > > > > be highlighted. > > > > > > > > > > > > > > > > > > > > 2. On release, this version (or it's dependencies) would > > not > > > > > have any > > > > > > > > > > > known MEDIUM/HIGH CVE. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This is a new policy and the details should be discussed. > > In > > > > > > > > particular, > > > > > > > > > > the threshold (medium or high). > > > > > > > > > > > > > > > > > > > > 3. Presence of any "early access"/"beta" feature should not > > > > > impact > > > > > > > > > > > other production features when it is not enabled. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This is a general guideline for early access features and > > not > > > > > specific > > > > > > > > to > > > > > > > > > > this release. It would be good to have a page that talks > > about > > > > > these > > > > > > > > > things. > > > > > > > > > > > > > > > > > > > > 4. Once enabled, users should have an option to disable any > > > > > "early > > > > > > > > > > > access"/"beta" feature and resume normal production > > features, > > > > > i.e. > > > > > > > > > > > impact of beta features should be reversible. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This needs discussion and I don't think it's reasonable as > > a > > > > > general > > > > > > > > > rule. > > > > > > > > > > For example, Kraft early access wasn't reversible and it > > was not > > > > > > > > feasible > > > > > > > > > > for it to be. > > > > > > > > > > > > > > > > > > > > 5. KIP-405 will be available in "early access"/"beta" > > mode. Early > > > > > > > > > > > access/beta means that the public facing interfaces won't > > > > > change in > > > > > > > > > > > future but the implementation is not recommended to be > > used in > > > > > > > > > > > production. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I don't think it's ok to make this a requirement. Early > > access > > > > > is a way > > > > > > > > > to > > > > > > > > > > get early feedback and all types of changes should be on > > the > > > > > table. > > > > > > > > They > > > > > > > > > > would be discussed via KIPs as usual. I believe there were > > some > > > > > > > > > > incompatible changes for Kraft during the early access > > period > > > > > although > > > > > > > > > the > > > > > > > > > > team aimed to minimize work required during upgrades. I > > have > > > > > mentioned > > > > > > > > > > Kraft a couple of times since it's a good example of a > > large > > > > > feature > > > > > > > > that > > > > > > > > > > went through this process. > > > > > > > > > > > > > > > > > > > > Ismael > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > Divij Vaidya > > > > > > >
