Hi Justine,
Adding KIP-890 part-1 to 3.6.0 seems reasonable to me. This part looks
to be addressing a critical issue of consumers getting stuck. Please
update the release plan wiki and merge all the required changes to 3.6
branch.

Thanks,
Satish.

On Thu, 24 Aug 2023 at 22:19, Justine Olshan
<jols...@confluent.io.invalid> wrote:
>
> Hey Satish,
> Does it make sense to include KIP-890 part 1? It prevents hanging
> transactions for older clients. (An optimization and stronger EOS
> guarantees will be included in part 2)
>
> Thanks,
> Justine
>
> On Mon, Aug 21, 2023 at 3:29 AM Satish Duggana <satish.dugg...@gmail.com>
> wrote:
>
> > Hi,
> > 3.6 branch is created. Please make sure any PRs targeted for 3.6.0
> > should be merged to 3.6 branch once those are merged to trunk.
> >
> > Thanks,
> > Satish.
> >
> > On Wed, 16 Aug 2023 at 15:58, Satish Duggana <satish.dugg...@gmail.com>
> > wrote:
> > >
> > > Hi,
> > > Please plan to merge PRs(including the major features) targeted for
> > > 3.6.0 by the end of Aug 20th UTC. Starting from August 21st, any pull
> > > requests intended for the 3.6.0 release must include the changes
> > > merged into the 3.6 branch as mentioned in the release plan.
> > >
> > > Thanks,
> > > Satish.
> > >
> > > On Fri, 4 Aug 2023 at 18:39, Chris Egerton <chr...@aiven.io.invalid>
> > wrote:
> > > >
> > > > Thanks for adding KIP-949, Satish!
> > > >
> > > > On Fri, Aug 4, 2023 at 7:06 AM Satish Duggana <
> > satish.dugg...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi,
> > > > > Myself and Divij discussed and added the wiki for Kafka TieredStorage
> > > > > Early Access Release[1]. If you have any comments or feedback, please
> > > > > feel free to share them.
> > > > >
> > > > > 1.
> > > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Tiered+Storage+Early+Access+Release+Notes
> > > > >
> > > > > Thanks,
> > > > > Satish.
> > > > >
> > > > > On Fri, 4 Aug 2023 at 08:40, Satish Duggana <
> > satish.dugg...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > Hi Chris,
> > > > > > Thanks for the update. This looks to be a minor change and is also
> > > > > > useful for backward compatibility. I added it to the release plan
> > as
> > > > > > an exceptional case.
> > > > > >
> > > > > > ~Satish.
> > > > > >
> > > > > > On Thu, 3 Aug 2023 at 21:34, Chris Egerton <chr...@aiven.io.invalid
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > Hi Satish,
> > > > > > >
> > > > > > > Would it be possible to include KIP-949 (
> > > > > > >
> > > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-949%3A+Add+flag+to+enable+the+usage+of+topic+separator+in+MM2+DefaultReplicationPolicy
> > > > > )
> > > > > > > in the 3.6.0 release? It passed voting yesterday, and is a very
> > small,
> > > > > > > low-risk change that we'd like to put out as soon as possible in
> > order
> > > > > to
> > > > > > > patch an accidental break in backwards compatibility caused a few
> > > > > versions
> > > > > > > ago.
> > > > > > >
> > > > > > > Best,
> > > > > > >
> > > > > > > Chris
> > > > > > >
> > > > > > > On Fri, Jul 28, 2023 at 2:35 AM Satish Duggana <
> > > > > satish.dugg...@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi All,
> > > > > > > > Whoever has KIP entries in the 3.6.0 release plan. Please
> > update it
> > > > > > > > with the latest status by tomorrow(end of the day 29th Jul UTC
> > ).
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > > Satish.
> > > > > > > >
> > > > > > > > On Fri, 28 Jul 2023 at 12:01, Satish Duggana <
> > > > > satish.dugg...@gmail.com>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Thanks Ismael and Divij for the suggestions.
> > > > > > > > >
> > > > > > > > > One way was to follow the earlier guidelines that we set for
> > any
> > > > > early
> > > > > > > > > access release. It looks Ismael already mentioned the
> > example of
> > > > > > > > > KRaft.
> > > > > > > > >
> > > > > > > > > KIP-405 mentions upgrade/downgrade and limitations sections.
> > We can
> > > > > > > > > clarify that in the release notes for users on how this
> > feature
> > > > > can be
> > > > > > > > > used for early access.
> > > > > > > > >
> > > > > > > > > Divij, We do not want users to enable this feature on
> > production
> > > > > > > > > environments in early access release. Let us work together
> > on the
> > > > > > > > > followups Ismael suggested.
> > > > > > > > >
> > > > > > > > > ~Satish.
> > > > > > > > >
> > > > > > > > > On Fri, 28 Jul 2023 at 02:24, Divij Vaidya <
> > > > > divijvaidy...@gmail.com>
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Those are great suggestions, thank you. We will continue
> > this
> > > > > > > > discussion
> > > > > > > > > > forward in a separate KIP for release plan for Tiered
> > Storage.
> > > > > > > > > >
> > > > > > > > > > On Thu 27. Jul 2023 at 21:46, Ismael Juma <
> > m...@ismaeljuma.com>
> > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Divij,
> > > > > > > > > > >
> > > > > > > > > > > I think the points you bring up for discussion are all
> > good.
> > > > > My main
> > > > > > > > > > > feedback is that they should be discussed in the context
> > of
> > > > > KIPs vs
> > > > > > > > the
> > > > > > > > > > > release template. That's why we have a backwards
> > compatibility
> > > > > > > > section for
> > > > > > > > > > > every KIP, it's precisely to ensure we think carefully
> > about
> > > > > some of
> > > > > > > > the
> > > > > > > > > > > points you're bringing up. When it comes to defining the
> > > > > meaning of
> > > > > > > > early
> > > > > > > > > > > access, we have two options:
> > > > > > > > > > >
> > > > > > > > > > > 1. Have a KIP specifically for tiered storage.
> > > > > > > > > > > 2. Have a KIP to define general guidelines for what early
> > > > > access
> > > > > > > > means.
> > > > > > > > > > >
> > > > > > > > > > > Does this make sense?
> > > > > > > > > > >
> > > > > > > > > > > Ismael
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Jul 27, 2023 at 6:38 PM Divij Vaidya <
> > > > > > > > divijvaidy...@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Thank you for the response, Ismael.
> > > > > > > > > > > >
> > > > > > > > > > > > 1. Specifically in context of 3.6, I wanted this
> > > > > compatibility
> > > > > > > > > > > > guarantee point to encourage a discussion on
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > >
> > > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-952%3A+Regenerate+segment-aligned+producer+snapshots+when+upgrading+to+a+Kafka+version+supporting+Tiered+Storage
> > > > > > > > > > > > .
> > > > > > > > > > > > Due to lack of producer snapshots in <2.8 versions, a
> > > > > customer may
> > > > > > > > not
> > > > > > > > > > > > be able to upgrade to 3.6 and use TS on a topic which
> > was
> > > > > created
> > > > > > > > when
> > > > > > > > > > > > the cluster was on <2.8 version (see motivation for
> > > > > details). We
> > > > > > > > can
> > > > > > > > > > > > discuss and agree that it does not break compatibility,
> > > > > which is
> > > > > > > > fine.
> > > > > > > > > > > > But I want to ensure that we have a discussion soon on
> > this
> > > > > to
> > > > > > > > reach a
> > > > > > > > > > > > conclusion.
> > > > > > > > > > > >
> > > > > > > > > > > > 2. I will start a KIP on this for further discussion.
> > > > > > > > > > > >
> > > > > > > > > > > > 3. In the context of 3.6, this would mean that there
> > should
> > > > > be
> > > > > > > > > > > > no-regression, if a user does "not" turn-on remote
> > storage
> > > > > (early
> > > > > > > > > > > > access feature) at a cluster level. We have some known
> > cases
> > > > > (such
> > > > > > > > as
> > > > > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-15189)
> > which
> > > > > violate
> > > > > > > > this
> > > > > > > > > > > > compatibility requirement. Having this guarantee
> > mentioned
> > > > > in the
> > > > > > > > > > > > release plan will ensure that we are all in agreement
> > with
> > > > > which
> > > > > > > > cases
> > > > > > > > > > > > are truly blockers and which aren't.
> > > > > > > > > > > >
> > > > > > > > > > > > 4. Fair, instead of a general goal, let me put it
> > > > > specifically in
> > > > > > > > the
> > > > > > > > > > > > context of 3.6. Let me know if this is not the right
> > forum
> > > > > for this
> > > > > > > > > > > > discussion.
> > > > > > > > > > > > Once a user "turns on" tiered storage (TS) at a cluster
> > > > > level, I am
> > > > > > > > > > > > proposing that they should have the ability to turn it
> > off
> > > > > as well
> > > > > > > > at
> > > > > > > > > > > > a cluster level. Since this is a topic level feature,
> > folks
> > > > > may not
> > > > > > > > > > > > spin up a separate cluster to try this feature, hence,
> > we
> > > > > need to
> > > > > > > > > > > > ensure that we provide them with the ability to try
> > tiered
> > > > > storage
> > > > > > > > for
> > > > > > > > > > > > a topic which could be deleted and featured
> > turned-off, so
> > > > > that
> > > > > > > > rest
> > > > > > > > > > > > of the production cases are not impacted.
> > > > > > > > > > > >
> > > > > > > > > > > > 5. Agree on not making public interface change as a
> > > > > requirement
> > > > > > > > but we
> > > > > > > > > > > > should define what "early access" means in that case.
> > Users
> > > > > may
> > > > > > > > not be
> > > > > > > > > > > > aware that "early access" public APIs may change
> > (unless I am
> > > > > > > > missing
> > > > > > > > > > > > some documentation somewhere completely, in which case
> > I
> > > > > apologize
> > > > > > > > for
> > > > > > > > > > > > bringing this naive point).
> > > > > > > > > > > >
> > > > > > > > > > > > --
> > > > > > > > > > > > Divij Vaidya
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Jul 27, 2023 at 2:27 PM Ismael Juma <
> > > > > m...@ismaeljuma.com>
> > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hi Divij,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Some of these are launch checklist items (not really
> > > > > goals) and
> > > > > > > > some
> > > > > > > > > > > are
> > > > > > > > > > > > > compatibility guarantees. More below.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, Jul 27, 2023, 12:10 PM Divij Vaidya <
> > > > > > > > divijvaidy...@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hey Satish
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Could we consider adding "launch goals" in the
> > release
> > > > > plan.
> > > > > > > > While
> > > > > > > > > > > > > > some of these may be implicit, it would be nice to
> > list
> > > > > them
> > > > > > > > down in
> > > > > > > > > > > > > > the release plan. For this release, our launch
> > > > > requirements
> > > > > > > > would be:
> > > > > > > > > > > > > > 1. Users should be able to upgrade from any prior
> > Kafka
> > > > > > > > version to
> > > > > > > > > > > this
> > > > > > > > > > > > > > version.
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > This is part of the compatibility guarantees. The
> > upgrade
> > > > > notes
> > > > > > > > mention
> > > > > > > > > > > > > this already. If there is a change in a given
> > release, it
> > > > > should
> > > > > > > > > > > > definitely
> > > > > > > > > > > > > be highlighted.
> > > > > > > > > > > > >
> > > > > > > > > > > > > 2. On release, this version (or it's dependencies)
> > would
> > > > > not
> > > > > > > > have any
> > > > > > > > > > > > > > known MEDIUM/HIGH CVE.
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > This is a new policy and the details should be
> > discussed.
> > > > > In
> > > > > > > > > > > particular,
> > > > > > > > > > > > > the threshold (medium or high).
> > > > > > > > > > > > >
> > > > > > > > > > > > > 3. Presence of any "early access"/"beta" feature
> > should not
> > > > > > > > impact
> > > > > > > > > > > > > > other production features when it is not enabled.
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > This is a general guideline for early access
> > features and
> > > > > not
> > > > > > > > specific
> > > > > > > > > > > to
> > > > > > > > > > > > > this release. It would be good to have a page that
> > talks
> > > > > about
> > > > > > > > these
> > > > > > > > > > > > things.
> > > > > > > > > > > > >
> > > > > > > > > > > > > 4. Once enabled, users should have an option to
> > disable any
> > > > > > > > "early
> > > > > > > > > > > > > > access"/"beta" feature and resume normal production
> > > > > features,
> > > > > > > > i.e.
> > > > > > > > > > > > > > impact of beta features should be reversible.
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > This needs discussion and I don't think it's
> > reasonable as
> > > > > a
> > > > > > > > general
> > > > > > > > > > > > rule.
> > > > > > > > > > > > > For example, Kraft early access wasn't reversible
> > and it
> > > > > was not
> > > > > > > > > > > feasible
> > > > > > > > > > > > > for it to be.
> > > > > > > > > > > > >
> > > > > > > > > > > > > 5. KIP-405 will be available in "early access"/"beta"
> > > > > mode. Early
> > > > > > > > > > > > > > access/beta means that the public facing
> > interfaces won't
> > > > > > > > change in
> > > > > > > > > > > > > > future but the implementation is not recommended
> > to be
> > > > > used in
> > > > > > > > > > > > > > production.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > I don't think it's ok to make this a requirement.
> > Early
> > > > > access
> > > > > > > > is a way
> > > > > > > > > > > > to
> > > > > > > > > > > > > get early feedback and all types of changes should
> > be on
> > > > > the
> > > > > > > > table.
> > > > > > > > > > > They
> > > > > > > > > > > > > would be discussed via KIPs as usual. I believe
> > there were
> > > > > some
> > > > > > > > > > > > > incompatible changes for Kraft during the early
> > access
> > > > > period
> > > > > > > > although
> > > > > > > > > > > > the
> > > > > > > > > > > > > team aimed to minimize work required during
> > upgrades. I
> > > > > have
> > > > > > > > mentioned
> > > > > > > > > > > > > Kraft a couple of times since it's a good example of
> > a
> > > > > large
> > > > > > > > feature
> > > > > > > > > > > that
> > > > > > > > > > > > > went through this process.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Ismael
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > Divij Vaidya
> > > > > > > >
> > > > >
> >

Reply via email to