Hi Justine, Adding KIP-890 part-1 to 3.6.0 seems reasonable to me. This part looks to be addressing a critical issue of consumers getting stuck. Please update the release plan wiki and merge all the required changes to 3.6 branch.
Thanks, Satish. On Thu, 24 Aug 2023 at 22:19, Justine Olshan <jols...@confluent.io.invalid> wrote: > > Hey Satish, > Does it make sense to include KIP-890 part 1? It prevents hanging > transactions for older clients. (An optimization and stronger EOS > guarantees will be included in part 2) > > Thanks, > Justine > > On Mon, Aug 21, 2023 at 3:29 AM Satish Duggana <satish.dugg...@gmail.com> > wrote: > > > Hi, > > 3.6 branch is created. Please make sure any PRs targeted for 3.6.0 > > should be merged to 3.6 branch once those are merged to trunk. > > > > Thanks, > > Satish. > > > > On Wed, 16 Aug 2023 at 15:58, Satish Duggana <satish.dugg...@gmail.com> > > wrote: > > > > > > Hi, > > > Please plan to merge PRs(including the major features) targeted for > > > 3.6.0 by the end of Aug 20th UTC. Starting from August 21st, any pull > > > requests intended for the 3.6.0 release must include the changes > > > merged into the 3.6 branch as mentioned in the release plan. > > > > > > Thanks, > > > Satish. > > > > > > On Fri, 4 Aug 2023 at 18:39, Chris Egerton <chr...@aiven.io.invalid> > > wrote: > > > > > > > > Thanks for adding KIP-949, Satish! > > > > > > > > On Fri, Aug 4, 2023 at 7:06 AM Satish Duggana < > > satish.dugg...@gmail.com> > > > > wrote: > > > > > > > > > Hi, > > > > > Myself and Divij discussed and added the wiki for Kafka TieredStorage > > > > > Early Access Release[1]. If you have any comments or feedback, please > > > > > feel free to share them. > > > > > > > > > > 1. > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Tiered+Storage+Early+Access+Release+Notes > > > > > > > > > > Thanks, > > > > > Satish. > > > > > > > > > > On Fri, 4 Aug 2023 at 08:40, Satish Duggana < > > satish.dugg...@gmail.com> > > > > > wrote: > > > > > > > > > > > > Hi Chris, > > > > > > Thanks for the update. This looks to be a minor change and is also > > > > > > useful for backward compatibility. I added it to the release plan > > as > > > > > > an exceptional case. > > > > > > > > > > > > ~Satish. > > > > > > > > > > > > On Thu, 3 Aug 2023 at 21:34, Chris Egerton <chr...@aiven.io.invalid > > > > > > > > wrote: > > > > > > > > > > > > > > Hi Satish, > > > > > > > > > > > > > > Would it be possible to include KIP-949 ( > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-949%3A+Add+flag+to+enable+the+usage+of+topic+separator+in+MM2+DefaultReplicationPolicy > > > > > ) > > > > > > > in the 3.6.0 release? It passed voting yesterday, and is a very > > small, > > > > > > > low-risk change that we'd like to put out as soon as possible in > > order > > > > > to > > > > > > > patch an accidental break in backwards compatibility caused a few > > > > > versions > > > > > > > ago. > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > > Chris > > > > > > > > > > > > > > On Fri, Jul 28, 2023 at 2:35 AM Satish Duggana < > > > > > satish.dugg...@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > Hi All, > > > > > > > > Whoever has KIP entries in the 3.6.0 release plan. Please > > update it > > > > > > > > with the latest status by tomorrow(end of the day 29th Jul UTC > > ). > > > > > > > > > > > > > > > > Thanks > > > > > > > > Satish. > > > > > > > > > > > > > > > > On Fri, 28 Jul 2023 at 12:01, Satish Duggana < > > > > > satish.dugg...@gmail.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > Thanks Ismael and Divij for the suggestions. > > > > > > > > > > > > > > > > > > One way was to follow the earlier guidelines that we set for > > any > > > > > early > > > > > > > > > access release. It looks Ismael already mentioned the > > example of > > > > > > > > > KRaft. > > > > > > > > > > > > > > > > > > KIP-405 mentions upgrade/downgrade and limitations sections. > > We can > > > > > > > > > clarify that in the release notes for users on how this > > feature > > > > > can be > > > > > > > > > used for early access. > > > > > > > > > > > > > > > > > > Divij, We do not want users to enable this feature on > > production > > > > > > > > > environments in early access release. Let us work together > > on the > > > > > > > > > followups Ismael suggested. > > > > > > > > > > > > > > > > > > ~Satish. > > > > > > > > > > > > > > > > > > On Fri, 28 Jul 2023 at 02:24, Divij Vaidya < > > > > > divijvaidy...@gmail.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > Those are great suggestions, thank you. We will continue > > this > > > > > > > > discussion > > > > > > > > > > forward in a separate KIP for release plan for Tiered > > Storage. > > > > > > > > > > > > > > > > > > > > On Thu 27. Jul 2023 at 21:46, Ismael Juma < > > m...@ismaeljuma.com> > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Hi Divij, > > > > > > > > > > > > > > > > > > > > > > I think the points you bring up for discussion are all > > good. > > > > > My main > > > > > > > > > > > feedback is that they should be discussed in the context > > of > > > > > KIPs vs > > > > > > > > the > > > > > > > > > > > release template. That's why we have a backwards > > compatibility > > > > > > > > section for > > > > > > > > > > > every KIP, it's precisely to ensure we think carefully > > about > > > > > some of > > > > > > > > the > > > > > > > > > > > points you're bringing up. When it comes to defining the > > > > > meaning of > > > > > > > > early > > > > > > > > > > > access, we have two options: > > > > > > > > > > > > > > > > > > > > > > 1. Have a KIP specifically for tiered storage. > > > > > > > > > > > 2. Have a KIP to define general guidelines for what early > > > > > access > > > > > > > > means. > > > > > > > > > > > > > > > > > > > > > > Does this make sense? > > > > > > > > > > > > > > > > > > > > > > Ismael > > > > > > > > > > > > > > > > > > > > > > On Thu, Jul 27, 2023 at 6:38 PM Divij Vaidya < > > > > > > > > divijvaidy...@gmail.com> > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Thank you for the response, Ismael. > > > > > > > > > > > > > > > > > > > > > > > > 1. Specifically in context of 3.6, I wanted this > > > > > compatibility > > > > > > > > > > > > guarantee point to encourage a discussion on > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-952%3A+Regenerate+segment-aligned+producer+snapshots+when+upgrading+to+a+Kafka+version+supporting+Tiered+Storage > > > > > > > > > > > > . > > > > > > > > > > > > Due to lack of producer snapshots in <2.8 versions, a > > > > > customer may > > > > > > > > not > > > > > > > > > > > > be able to upgrade to 3.6 and use TS on a topic which > > was > > > > > created > > > > > > > > when > > > > > > > > > > > > the cluster was on <2.8 version (see motivation for > > > > > details). We > > > > > > > > can > > > > > > > > > > > > discuss and agree that it does not break compatibility, > > > > > which is > > > > > > > > fine. > > > > > > > > > > > > But I want to ensure that we have a discussion soon on > > this > > > > > to > > > > > > > > reach a > > > > > > > > > > > > conclusion. > > > > > > > > > > > > > > > > > > > > > > > > 2. I will start a KIP on this for further discussion. > > > > > > > > > > > > > > > > > > > > > > > > 3. In the context of 3.6, this would mean that there > > should > > > > > be > > > > > > > > > > > > no-regression, if a user does "not" turn-on remote > > storage > > > > > (early > > > > > > > > > > > > access feature) at a cluster level. We have some known > > cases > > > > > (such > > > > > > > > as > > > > > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-15189) > > which > > > > > violate > > > > > > > > this > > > > > > > > > > > > compatibility requirement. Having this guarantee > > mentioned > > > > > in the > > > > > > > > > > > > release plan will ensure that we are all in agreement > > with > > > > > which > > > > > > > > cases > > > > > > > > > > > > are truly blockers and which aren't. > > > > > > > > > > > > > > > > > > > > > > > > 4. Fair, instead of a general goal, let me put it > > > > > specifically in > > > > > > > > the > > > > > > > > > > > > context of 3.6. Let me know if this is not the right > > forum > > > > > for this > > > > > > > > > > > > discussion. > > > > > > > > > > > > Once a user "turns on" tiered storage (TS) at a cluster > > > > > level, I am > > > > > > > > > > > > proposing that they should have the ability to turn it > > off > > > > > as well > > > > > > > > at > > > > > > > > > > > > a cluster level. Since this is a topic level feature, > > folks > > > > > may not > > > > > > > > > > > > spin up a separate cluster to try this feature, hence, > > we > > > > > need to > > > > > > > > > > > > ensure that we provide them with the ability to try > > tiered > > > > > storage > > > > > > > > for > > > > > > > > > > > > a topic which could be deleted and featured > > turned-off, so > > > > > that > > > > > > > > rest > > > > > > > > > > > > of the production cases are not impacted. > > > > > > > > > > > > > > > > > > > > > > > > 5. Agree on not making public interface change as a > > > > > requirement > > > > > > > > but we > > > > > > > > > > > > should define what "early access" means in that case. > > Users > > > > > may > > > > > > > > not be > > > > > > > > > > > > aware that "early access" public APIs may change > > (unless I am > > > > > > > > missing > > > > > > > > > > > > some documentation somewhere completely, in which case > > I > > > > > apologize > > > > > > > > for > > > > > > > > > > > > bringing this naive point). > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > Divij Vaidya > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Jul 27, 2023 at 2:27 PM Ismael Juma < > > > > > m...@ismaeljuma.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Divij, > > > > > > > > > > > > > > > > > > > > > > > > > > Some of these are launch checklist items (not really > > > > > goals) and > > > > > > > > some > > > > > > > > > > > are > > > > > > > > > > > > > compatibility guarantees. More below. > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Jul 27, 2023, 12:10 PM Divij Vaidya < > > > > > > > > divijvaidy...@gmail.com> > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > Hey Satish > > > > > > > > > > > > > > > > > > > > > > > > > > > > Could we consider adding "launch goals" in the > > release > > > > > plan. > > > > > > > > While > > > > > > > > > > > > > > some of these may be implicit, it would be nice to > > list > > > > > them > > > > > > > > down in > > > > > > > > > > > > > > the release plan. For this release, our launch > > > > > requirements > > > > > > > > would be: > > > > > > > > > > > > > > 1. Users should be able to upgrade from any prior > > Kafka > > > > > > > > version to > > > > > > > > > > > this > > > > > > > > > > > > > > version. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This is part of the compatibility guarantees. The > > upgrade > > > > > notes > > > > > > > > mention > > > > > > > > > > > > > this already. If there is a change in a given > > release, it > > > > > should > > > > > > > > > > > > definitely > > > > > > > > > > > > > be highlighted. > > > > > > > > > > > > > > > > > > > > > > > > > > 2. On release, this version (or it's dependencies) > > would > > > > > not > > > > > > > > have any > > > > > > > > > > > > > > known MEDIUM/HIGH CVE. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This is a new policy and the details should be > > discussed. > > > > > In > > > > > > > > > > > particular, > > > > > > > > > > > > > the threshold (medium or high). > > > > > > > > > > > > > > > > > > > > > > > > > > 3. Presence of any "early access"/"beta" feature > > should not > > > > > > > > impact > > > > > > > > > > > > > > other production features when it is not enabled. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This is a general guideline for early access > > features and > > > > > not > > > > > > > > specific > > > > > > > > > > > to > > > > > > > > > > > > > this release. It would be good to have a page that > > talks > > > > > about > > > > > > > > these > > > > > > > > > > > > things. > > > > > > > > > > > > > > > > > > > > > > > > > > 4. Once enabled, users should have an option to > > disable any > > > > > > > > "early > > > > > > > > > > > > > > access"/"beta" feature and resume normal production > > > > > features, > > > > > > > > i.e. > > > > > > > > > > > > > > impact of beta features should be reversible. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This needs discussion and I don't think it's > > reasonable as > > > > > a > > > > > > > > general > > > > > > > > > > > > rule. > > > > > > > > > > > > > For example, Kraft early access wasn't reversible > > and it > > > > > was not > > > > > > > > > > > feasible > > > > > > > > > > > > > for it to be. > > > > > > > > > > > > > > > > > > > > > > > > > > 5. KIP-405 will be available in "early access"/"beta" > > > > > mode. Early > > > > > > > > > > > > > > access/beta means that the public facing > > interfaces won't > > > > > > > > change in > > > > > > > > > > > > > > future but the implementation is not recommended > > to be > > > > > used in > > > > > > > > > > > > > > production. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I don't think it's ok to make this a requirement. > > Early > > > > > access > > > > > > > > is a way > > > > > > > > > > > > to > > > > > > > > > > > > > get early feedback and all types of changes should > > be on > > > > > the > > > > > > > > table. > > > > > > > > > > > They > > > > > > > > > > > > > would be discussed via KIPs as usual. I believe > > there were > > > > > some > > > > > > > > > > > > > incompatible changes for Kraft during the early > > access > > > > > period > > > > > > > > although > > > > > > > > > > > > the > > > > > > > > > > > > > team aimed to minimize work required during > > upgrades. I > > > > > have > > > > > > > > mentioned > > > > > > > > > > > > > Kraft a couple of times since it's a good example of > > a > > > > > large > > > > > > > > feature > > > > > > > > > > > that > > > > > > > > > > > > > went through this process. > > > > > > > > > > > > > > > > > > > > > > > > > > Ismael > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > Divij Vaidya > > > > > > > > > > > > > > >
