Given the discussed upgrade path from the current join semantics to the improved join semantics, I think it's best to partly revert KIP-633 for 3.0 release.
I prepared a PR for it: https://github.com/apache/kafka/pull/11233 The idea is to stay on `JoinWindows.of().grace()` and not add `JoinWindows.ofTimeDifferenceAndGrace / NoGrace()`: this way users cannot enable the spurious left/outer join fix and we are safe. We can fix the join bug and complete KIP-633 in 3.1 release. As we undo the deprecation for `JoinWindows.of()` and `JoinWindows.grace()`, I assume your talk is still correct... I actually also talked about the stream/stream join fix in Kafka Summit Europe this yeer "advertising" the improved join semantics... So we are somewhat in the same boat. Still think it's fine if it only ships in 3.1 instead of 3.0. I understand that it is annoying to only ship a partially implemented KIP, but I strongly believe it's the best way forward. -Matthias On 8/19/21 2:53 PM, Israel Ekpo wrote: > Konstantine, > > After a brief chat with Sophie, I just updated the name for KIP-633 to be > more descriptive of what is actually happening in the implementation > > It is changed on the KIP page and the JIRA task title is also updated to be > more descriptive > https://cwiki.apache.org/confluence/x/Ho2NCg > https://issues.apache.org/jira/browse/KAFKA-8613 > > However, I could not update the page for the 3.0 release plan. I do not > have the permissions for that update. > https://cwiki.apache.org/confluence/x/woONCg > > When you have a moment, please could you update the release plan for 3.0 to > reflect the update name for the KIP? > > Thank you very much > > Sincerely, > Israel > > > On Thu, Aug 19, 2021 at 5:50 PM Israel Ekpo <israele...@gmail.com> wrote: > >> Hi Matthias, it is possible to still deprecate JoinWindows.of(size) even >> though the new join semantics is disabled. >> >> I just pre-recorded a talk for Kafka Summit Americas where I am >> recommending a switch to the new APIs instead of the deprecated one >> starting from 3.0 >> >> I would love to be involved in the discussion for the fix so that we try >> to honor the expectations of the KIP as much as possible. >> >> So far, it looks like the PR will still honor the expectations of the KIP >> >> https://github.com/apache/kafka/pull/11235 >> >> Thanks for sharing this to create awareness. >> >> Sincerely, >> Israel >> >> >> >> >> >> On Wed, Aug 18, 2021 at 8:57 PM Matthias J. Sax <mj...@apache.org> wrote: >> >>> Hi, >>> >>> we discovered a potential blocker for 3.0.0 today: >>> https://issues.apache.org/jira/browse/KAFKA-13216 >>> >>> We are still evaluating a potential fix. If we cannot fix it quickly, >>> the fall-back would be to partially roll-back KIP-633, to disable the >>> new join semantics such that people cannot hit this bug. >>> >>> Thoughts? >>> >>> >>> -Matthias >>> >>> >>> On 8/17/21 1:29 PM, Konstantine Karantasis wrote: >>>> Hi Rajini. >>>> >>>> Approved, given its low risk and the lack of convenient workarounds. >>>> >>>> Konstantine >>>> >>>> >>>> On Tue, Aug 17, 2021 at 11:00 AM Rajini Sivaram < >>> rajinisiva...@gmail.com> >>>> wrote: >>>> >>>>> Hi Konstantine, >>>>> >>>>> We found an issue with replication with IBP 2.7: >>>>> https://issues.apache.org/jira/browse/KAFKA-13207. The fix is small >>> and >>>>> low >>>>> risk and has been merged to trunk. Can we include this in 3.0 branch >>> since >>>>> it can result in IllegalStateException during replication? >>>>> >>>>> Thank you, >>>>> >>>>> Rajini >>>>> >>>>> >>>>> On Thu, Aug 5, 2021 at 6:21 PM Konstantine Karantasis < >>>>> kkaranta...@apache.org> wrote: >>>>> >>>>>> Jose, thanks for the heads up on the 3 new blocker candidates. >>>>>> >>>>>> I read the tickets and they have clear descriptions and implementation >>>>>> details. >>>>>> However, at this stage to be able to make a call and approve new >>> blockers >>>>>> I'd appreciate it if we could get some insight regarding the risk and >>> the >>>>>> necessity of a fix. A rough ETA would also be helpful. >>>>>> >>>>>> Having said that, based on the descriptions and the existence of a few >>>>>> other blockers, I'm tentatively approving KAFKA-13161, KAFKA-13165, >>> and >>>>>> KAFKA-13168 and we might have to make a new assessment if these are >>> the >>>>>> only blockers in the next few days or if we notice a regression during >>>>>> testing. >>>>>> >>>>>> Konstantine >>>>>> >>>>>> >>>>>> >>>>>> On Thu, Aug 5, 2021 at 10:04 AM Konstantine Karantasis < >>>>>> kkaranta...@apache.org> wrote: >>>>>> >>>>>>> >>>>>>> Thanks for reporting this new issue Ryan, >>>>>>> >>>>>>> It's important and this issue seems to have clearly regressed dynamic >>>>>>> default configs in the 3.0 branch. >>>>>>> So, it's approved. >>>>>>> >>>>>>> Konstantine >>>>>>> >>>>>>> >>>>>>> On Wed, Aug 4, 2021 at 4:34 PM José Armando García Sancio >>>>>>> <jsan...@confluent.io.invalid> wrote: >>>>>>> >>>>>>>> Hey all, >>>>>>>> >>>>>>>> For the KIP-500 work for 3.0 we would like to propose the following >>>>>>>> Jiras as blockers: >>>>>>>> >>>>>>>> 1. https://issues.apache.org/jira/browse/KAFKA-13168 >>>>>>>> 2. https://issues.apache.org/jira/browse/KAFKA-13165 >>>>>>>> 3. https://issues.apache.org/jira/browse/KAFKA-13161 >>>>>>>> >>>>>>>> The description for each Jira should have more details. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> -Jose >>>>>>>> >>>>>>>> On Tue, Aug 3, 2021 at 12:14 PM Ryan Dielhenn >>>>>>>> <rdielh...@confluent.io.invalid> wrote: >>>>>>>>> >>>>>>>>> Hi Konstantine, >>>>>>>>> >>>>>>>>> I would like to report another bug in KRaft. >>>>>>>>> >>>>>>>>> The ConfigHandler that processes dynamic broker config deltas in >>>>> KRaft >>>>>>>>> expects that the default resource name for dynamic broker configs >>> is >>>>>> the >>>>>>>>> old default entity name used in ZK: "<default>". Since dynamic >>>>> default >>>>>>>>> broker configs are persisted as empty string in the quorum instead >>>>> of >>>>>>>>> "<default>", the brokers are not updating the their default >>>>>>>> configuration >>>>>>>>> when they see empty string as a resource name in the config delta >>>>> and >>>>>>>> are >>>>>>>>> throwing a NumberFormatException when they try to parse the >>> resource >>>>>>>> name >>>>>>>>> to process it as a per-broker configuration. >>>>>>>>> >>>>>>>>> I filed a JIRA: https://issues.apache.org/jira/browse/KAFKA-13160 >>>>>>>>> >>>>>>>>> I also have a PR to fix this: >>>>>>>> https://github.com/apache/kafka/pull/11168 >>>>>>>>> >>>>>>>>> I think that this should be a blocker for 3.0 because dynamic >>>>> default >>>>>>>>> broker configs will not be usable in KRaft otherwise. >>>>>>>>> >>>>>>>>> Best, >>>>>>>>> Ryan Dielhenn >>>>>>>>> >>>>>>>>> On Sat, Jul 31, 2021 at 10:42 AM Konstantine Karantasis < >>>>>>>>> kkaranta...@apache.org> wrote: >>>>>>>>> >>>>>>>>>> Thanks Ryan, >>>>>>>>>> >>>>>>>>>> Approved. Seems also like a low risk fix. >>>>>>>>>> With that opportunity, let's make sure there are no other configs >>>>>> that >>>>>>>>>> would need a similar validation. >>>>>>>>>> >>>>>>>>>> Konstantine >>>>>>>>>> >>>>>>>>>> On Fri, Jul 30, 2021 at 8:33 AM Ryan Dielhenn >>>>>>>>>> <rdielh...@confluent.io.invalid> wrote: >>>>>>>>>> >>>>>>>>>>> Hey Konstantine, >>>>>>>>>>> >>>>>>>>>>> Thanks for the question. If these configs are not validated the >>>>>>>> user's >>>>>>>>>>> experience will be affected and upgrades from 3.0 will be >>>>> harder. >>>>>>>>>>> >>>>>>>>>>> Best, >>>>>>>>>>> Ryan Dielhenn >>>>>>>>>>> >>>>>>>>>>> On Thu, Jul 29, 2021 at 3:59 PM Konstantine Karantasis < >>>>>>>>>>> kkaranta...@apache.org> wrote: >>>>>>>>>>> >>>>>>>>>>>> Thanks for reporting this issue Ryan. >>>>>>>>>>>> >>>>>>>>>>>> I believe what you mention corresponds to the ticket you >>>>> created >>>>>>>> here: >>>>>>>>>>>> >>>>>> https://issues.apache.org/jira/projects/KAFKA/issues/KAFKA-13151 >>>>>>>>>>>> >>>>>>>>>>>> What happens if the configurations are present but the broker >>>>>>>> doesn't >>>>>>>>>>> fail >>>>>>>>>>>> at startup when configured to run in KRaft mode? >>>>>>>>>>>> Asking to see if we have any workarounds in our availability. >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> Konstantine >>>>>>>>>>>> >>>>>>>>>>>> On Thu, Jul 29, 2021 at 2:51 PM Ryan Dielhenn >>>>>>>>>>>> <rdielh...@confluent.io.invalid> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi, >>>>>>>>>>>>> >>>>>>>>>>>>> Disregard log.clean.policy being included in this blocker. >>>>>>>>>>>>> >>>>>>>>>>>>> Best, >>>>>>>>>>>>> Ryan Dielhenn >>>>>>>>>>>>> >>>>>>>>>>>>> On Thu, Jul 29, 2021 at 2:38 PM Ryan Dielhenn < >>>>>>>>>> rdielh...@confluent.io> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hey Konstantine, >>>>>>>>>>>>>> >>>>>>>>>>>>>> I'd like to report another bug in KRaft. >>>>>>>>>>>>>> >>>>>>>>>>>>>> log.cleanup.policy, alter.config.policy.class.name, and >>>>>>>>>>>>>> create.topic.policy.class.name are all unsupported by >>>>> KRaft >>>>>>>> but >>>>>>>>>>> KRaft >>>>>>>>>>>>>> servers allow them to be configured. I believe this should >>>>>> be >>>>>>>>>>>> considered >>>>>>>>>>>>> a >>>>>>>>>>>>>> blocker and that KRaft servers should fail startup if any >>>>> of >>>>>>>> these >>>>>>>>>>> are >>>>>>>>>>>>>> configured. I do not have a PR yet but will soon. >>>>>>>>>>>>>> >>>>>>>>>>>>>> On another note, I have a PR for the dynamic broker >>>>>>>> configuration >>>>>>>>>> fix >>>>>>>>>>>>>> here: https://github.com/apache/kafka/pull/11141 >>>>>>>>>>>>>> >>>>>>>>>>>>>> Best, >>>>>>>>>>>>>> Ryan Dielhenn >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Wed, May 26, 2021 at 2:48 PM Konstantine Karantasis >>>>>>>>>>>>>> <konstant...@confluent.io.invalid> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi all, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Please find below the updated release plan for the Apache >>>>>>>> Kafka >>>>>>>>>>> 3.0.0 >>>>>>>>>>>>>>> release. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>> >>>>> >>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177046466 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> New suggested dates for the release are as follows: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> KIP Freeze is 09 June 2021 (same date as in the initial >>>>>> plan) >>>>>>>>>>>>>>> Feature Freeze is 30 June 2021 (new date, extended by two >>>>>>>> weeks) >>>>>>>>>>>>>>> Code Freeze is 14 July 2021 (new date, extended by two >>>>>> weeks) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> At least two weeks of stabilization will follow Code >>>>>> Freeze. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> The release plan is up to date and currently includes all >>>>>> the >>>>>>>>>>> approved >>>>>>>>>>>>>>> KIPs >>>>>>>>>>>>>>> that are targeting 3.0.0. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Please let me know if you have any objections with the >>>>>> recent >>>>>>>>>>>> extension >>>>>>>>>>>>> of >>>>>>>>>>>>>>> Feature Freeze and Code Freeze or any other concerns. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>> Konstantine >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> -Jose >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
