Hey Divij, that is a good point regarding KIP-848. David, as the author of the KIP, would you be able to drive this?
Similarly, would anybody be willing to drive such an EA Release Note for the JBOD feature? Mayank, this *doesn't* seem like a blocker to me given the complexity, public API changes and the fact that it's for high partition counts (a definition here would help though) Reminder - RC4 is out for vote right now. Best, Stanislav On Tue, Feb 6, 2024 at 5:25 PM Mayank Shekhar Narula < mayanks.nar...@gmail.com> wrote: > Hi Folks > > KIP-951 was delivered fully in AK 3.7. Its 1st optimisation was delivered > in 3.6.1, to skip backoff period for a produce batch being retried to new > leader i.e. KAFKA-15415. > > KAFKA-15415 current implementation introduced a performance regression, by > increasing synchronization on the produce path, especially for high > partition counts. The description section of > https://issues.apache.org/jira/browse/KAFKA-16226 goes more into details > of > the regression. > > I have put up a fix https://github.com/apache/kafka/pull/15323, which > removes this synchronization. The fix adds a new public method to > Cluster.java, and a public constructor to PartitionInfo.java. > > Is this a blocker for v3.7.0? > > PS - Posted in KIP-951's voting thread as well > <https://lists.apache.org/thread/otxt5wr7cj4qx4v3zg05gclry0vrdvh8>. > > > On Fri, Feb 2, 2024 at 3:58 PM Divij Vaidya <divijvaidy...@gmail.com> > wrote: > > > Hey folks > > > > The release plan for 3.7.0 [1] calls out KIP 848 as "Targeting a Preview > in > > 3.7". > > > > Is that still true? If yes, then we should perhaps add that in the blog, > > call it out in the release notes and prepare a preview document similar > to > > what we did for Tiered Storage Early Access release[2] > > > > If not true, then we should update the release notes to reflect the > current > > state of the KIP. > > > > (I think the same is true for other KIPs like KIP-963) > > > > [1] https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.7.0 > > [2] > > > > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Tiered+Storage+Early+Access+Release+Notes > > > > > > -- > > Divij Vaidya > > > > > > > > On Thu, Jan 11, 2024 at 1:03 PM Luke Chen <show...@gmail.com> wrote: > > > > > Hi all, > > > > > > There is a bug KAFKA-16101 > > > <https://issues.apache.org/jira/browse/KAFKA-16101> reporting that > > "Kafka > > > cluster will be unavailable during KRaft migration rollback". > > > The impact for this issue is that if brokers try to rollback to ZK mode > > > during KRaft migration process, there will be a period of time the > > cluster > > > is unavailable. > > > Since ZK migrating to KRaft feature is a production ready feature, I > > think > > > this should be addressed soon. > > > Do you think this is a blocker for v3.7.0? > > > > > > Thanks. > > > Luke > > > > > > On Thu, Jan 11, 2024 at 6:11 AM Stanislav Kozlovski > > > <stanis...@confluent.io.invalid> wrote: > > > > > > > Thanks Colin, > > > > > > > > With that, I believe we are out of blockers. I was traveling today > and > > > > couldn't build an RC - expect one to be published tomorrow (barring > any > > > > problems). > > > > > > > > In the meanwhile - here is a PR for the 3.7 blog post - > > > > https://github.com/apache/kafka-site/pull/578 > > > > > > > > Best, > > > > Stan > > > > > > > > On Wed, Jan 10, 2024 at 12:06 AM Colin McCabe <cmcc...@apache.org> > > > wrote: > > > > > > > > > KAFKA-16094 has been fixed and backported to 3.7. > > > > > > > > > > Colin > > > > > > > > > > > > > > > On Mon, Jan 8, 2024, at 14:52, Colin McCabe wrote: > > > > > > On an unrelated note, I found a blocker bug related to upgrades > > from > > > > > > 3.6 (and earlier) to 3.7. > > > > > > > > > > > > The JIRA is here: > > > > > > https://issues.apache.org/jira/browse/KAFKA-16094 > > > > > > > > > > > > Fix here: > > > > > > https://github.com/apache/kafka/pull/15153 > > > > > > > > > > > > best, > > > > > > Colin > > > > > > > > > > > > > > > > > > On Mon, Jan 8, 2024, at 14:47, Colin McCabe wrote: > > > > > >> Hi Ismael, > > > > > >> > > > > > >> I wasn't aware of that. If we are required to publish all > modules, > > > > then > > > > > >> this is working as intended. > > > > > >> > > > > > >> I am a bit curious if we've discussed why we need to publish the > > > > server > > > > > >> modules to Sonatype. Is there a discussion about the pros and > cons > > > of > > > > > >> this somewhere? > > > > > >> > > > > > >> regards, > > > > > >> Colin > > > > > >> > > > > > >> On Mon, Jan 8, 2024, at 14:09, Ismael Juma wrote: > > > > > >>> All modules are published to Sonatype - that's a requirement. > You > > > may > > > > > be > > > > > >>> missing the fact that `core` is published as `kafka_2.13` and > > > > > `kafka_2.12`. > > > > > >>> > > > > > >>> Ismael > > > > > >>> > > > > > >>> On Tue, Jan 9, 2024 at 12:00 AM Colin McCabe < > cmcc...@apache.org > > > > > > > > wrote: > > > > > >>> > > > > > >>>> Hi Ismael, > > > > > >>>> > > > > > >>>> It seems like both the metadata gradle module and the > > > server-common > > > > > module > > > > > >>>> are getting published to Sonatype as separate artifacts, > unless > > > I'm > > > > > >>>> misunderstanding something. Example: > > > > > >>>> > > > > > >>>> https://central.sonatype.com/search?q=kafka-server-common > > > > > >>>> > > > > > >>>> I don't see kafka-core getting published, but maybe other > > private > > > > > >>>> server-side gradle modules are getting published. > > > > > >>>> > > > > > >>>> This seems bad. Is there a reason to publish modules that are > > only > > > > > used by > > > > > >>>> the server on Sonatype? > > > > > >>>> > > > > > >>>> best, > > > > > >>>> Colin > > > > > >>>> > > > > > >>>> > > > > > >>>> On Mon, Jan 8, 2024, at 12:50, Ismael Juma wrote: > > > > > >>>> > Hi Colin, > > > > > >>>> > > > > > > >>>> > I think you may have misunderstood what they mean by gradle > > > > > metadata - > > > > > >>>> it's > > > > > >>>> > not the Kafka metadata module. > > > > > >>>> > > > > > > >>>> > Ismael > > > > > >>>> > > > > > > >>>> > On Mon, Jan 8, 2024 at 9:45 PM Colin McCabe < > > cmcc...@apache.org > > > > > > > > > wrote: > > > > > >>>> > > > > > > >>>> >> Oops, hit send too soon. I see that #15127 was already > > merged. > > > So > > > > > we > > > > > >>>> >> should no longer be publishing :metadata as part of the > > clients > > > > > >>>> artifacts, > > > > > >>>> >> right? > > > > > >>>> >> > > > > > >>>> >> thanks, > > > > > >>>> >> Colin > > > > > >>>> >> > > > > > >>>> >> > > > > > >>>> >> On Mon, Jan 8, 2024, at 11:42, Colin McCabe wrote: > > > > > >>>> >> > Hi Apporv, > > > > > >>>> >> > > > > > > >>>> >> > Please remove the metadata module from any artifacts > > > published > > > > > for > > > > > >>>> >> > clients. It is only used by the server. > > > > > >>>> >> > > > > > > >>>> >> > best, > > > > > >>>> >> > Colin > > > > > >>>> >> > > > > > > >>>> >> > > > > > > >>>> >> > On Sun, Jan 7, 2024, at 03:04, Apoorv Mittal wrote: > > > > > >>>> >> >> Hi Colin, > > > > > >>>> >> >> Thanks for the response. The only reason for asking the > > > > > question of > > > > > >>>> >> >> publishing the metadata is because that's present in > > > previous > > > > > client > > > > > >>>> >> >> releases. For more context, the description of PR > > > > > >>>> >> >> <https://github.com/apache/kafka/pull/15127> holds the > > > > details > > > > > and > > > > > >>>> >> waiting > > > > > >>>> >> >> for the confirmation there prior to the merge. > > > > > >>>> >> >> > > > > > >>>> >> >> Regards, > > > > > >>>> >> >> Apoorv Mittal > > > > > >>>> >> >> +44 7721681581 > > > > > >>>> >> >> > > > > > >>>> >> >> > > > > > >>>> >> >> On Fri, Jan 5, 2024 at 10:22 PM Colin McCabe < > > > > > cmcc...@apache.org> > > > > > >>>> >> wrote: > > > > > >>>> >> >> > > > > > >>>> >> >>> metadata is an internal gradle module. It is not used > by > > > > > clients. > > > > > >>>> So I > > > > > >>>> >> >>> don't see why you would want to publish it (unless I'm > > > > > >>>> misunderstanding > > > > > >>>> >> >>> something). > > > > > >>>> >> >>> > > > > > >>>> >> >>> best, > > > > > >>>> >> >>> Colin > > > > > >>>> >> >>> > > > > > >>>> >> >>> > > > > > >>>> >> >>> On Fri, Jan 5, 2024, at 10:05, Stanislav Kozlovski > wrote: > > > > > >>>> >> >>> > Thanks for reporting the blockers, folks. Good job > > > finding. > > > > > >>>> >> >>> > > > > > > >>>> >> >>> > I have one ask - can anybody with Gradle expertise > help > > > > > review > > > > > >>>> this > > > > > >>>> >> small > > > > > >>>> >> >>> > PR? https://github.com/apache/kafka/pull/15127 (+1, > > -1) > > > > > >>>> >> >>> > In particular, we are wondering whether we need to > > > publish > > > > > module > > > > > >>>> >> >>> metadata > > > > > >>>> >> >>> > as part of the gradle publishing process. > > > > > >>>> >> >>> > > > > > > >>>> >> >>> > > > > > > >>>> >> >>> > On Fri, Jan 5, 2024 at 3:56 PM Proven Provenzano > > > > > >>>> >> >>> > <pprovenz...@confluent.io.invalid> wrote: > > > > > >>>> >> >>> > > > > > > >>>> >> >>> >> We have potentially one more blocker > > > > > >>>> >> >>> >> https://issues.apache.org/jira/browse/KAFKA-16082 > > which > > > > > might > > > > > >>>> >> cause a > > > > > >>>> >> >>> data > > > > > >>>> >> >>> >> loss scenario with JBOD in KRaft. > > > > > >>>> >> >>> >> Initial analysis thought this is a problem and > further > > > > > review > > > > > >>>> looks > > > > > >>>> >> >>> like it > > > > > >>>> >> >>> >> isn't but we are continuing to dig into the issue to > > > > ensure > > > > > that > > > > > >>>> it > > > > > >>>> >> >>> isn't. > > > > > >>>> >> >>> >> We would request feedback on the bug from anyone who > > is > > > > > familiar > > > > > >>>> >> with > > > > > >>>> >> >>> this > > > > > >>>> >> >>> >> code. > > > > > >>>> >> >>> >> > > > > > >>>> >> >>> >> --Proven > > > > > >>>> >> >>> >> > > > > > >>>> >> >>> > > > > > > >>>> >> >>> > > > > > > >>>> >> >>> > -- > > > > > >>>> >> >>> > Best, > > > > > >>>> >> >>> > Stanislav > > > > > >>>> >> >>> > > > > > >>>> >> > > > > > >>>> > > > > > > > > > > > > > > > > > -- > > > > Best, > > > > Stanislav > > > > > > > > > > > > -- > Regards, > Mayank Shekhar Narula > -- Best, Stanislav
