Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #2551

[Apache Jenkins Server]

See 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.7 #53

[Apache Jenkins Server]

See 

[jira] [Resolved] (KAFKA-15250) DefaultBackgroundThread is running tight loop

[Philip Nee (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Philip Nee resolved KAFKA-15250.

Fix Version/s: (was: 3.8.0)
   Resolution: Fixed

> DefaultBackgroundThread is running tight loop
> -
>
> Key: KAFKA-15250
> URL: https://issues.apache.org/jira/browse/KAFKA-15250
> Project: Kafka
>  Issue Type: Bug
>  Components: clients, consumer
>Reporter: Philip Nee
>Assignee: Philip Nee
>Priority: Major
>  Labels: consumer-threading-refactor
>
> The DefaultBackgroundThread is running tight loops and wasting CPU cycles.  I 
> think we need to reexamine the timeout pass to networkclientDelegate.poll.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2550

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 457704 lines...]

Gradle Test Run :streams:test > Gradle Test Executor 91 > TasksTest > 
onlyRemovePendingTaskToCloseReviveAndUpdateInputPartitionsShouldRemoveTaskFromPendingUpdateActions()
 PASSED

Gradle Test Run :streams:test > Gradle Test Executor 91 > TasksTest > 
shouldVerifyIfPendingTaskToInitExist() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 91 > TasksTest > 
shouldVerifyIfPendingTaskToInitExist() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 91 > TasksTest > 
shouldAddAndRemovePendingTaskToAddBack() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 91 > TasksTest > 
shouldAddAndRemovePendingTaskToAddBack() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 91 > TasksTest > 
onlyRemovePendingTaskToCloseCleanShouldRemoveTaskFromPendingUpdateActions() 
STARTED

Gradle Test Run :streams:test > Gradle Test Executor 91 > TasksTest > 
onlyRemovePendingTaskToCloseCleanShouldRemoveTaskFromPendingUpdateActions() 
PASSED

Gradle Test Run :streams:test > Gradle Test Executor 91 > TasksTest > 
shouldCheckStateWhenRemoveTask() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 91 > TasksTest > 
shouldCheckStateWhenRemoveTask() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 91 > TasksTest > 
shouldDrainPendingTasksToCreate() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 91 > TasksTest > 
shouldDrainPendingTasksToCreate() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 91 > TasksTest > 
onlyRemovePendingTaskToRecycleShouldRemoveTaskFromPendingUpdateActions() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 91 > TasksTest > 
onlyRemovePendingTaskToRecycleShouldRemoveTaskFromPendingUpdateActions() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 91 > TasksTest > 
shouldAddAndRemovePendingTaskToCloseClean() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 91 > TasksTest > 
shouldAddAndRemovePendingTaskToCloseClean() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 91 > TasksTest > 
shouldKeepAddedTasks() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 91 > TasksTest > 
shouldKeepAddedTasks() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 91 > 
RocksDBBlockCacheMetricsTest > shouldRecordCorrectBlockCacheUsage(RocksDBStore, 
StateStoreContext) > "shouldRecordCorrectBlockCacheUsage(RocksDBStore, 
StateStoreContext).org.apache.kafka.streams.state.internals.RocksDBStore@21450010,
 org.apache.kafka.test.MockInternalProcessorContext@6b9155cf" STARTED

Gradle Test Run :streams:test > Gradle Test Executor 91 > 
RocksDBBlockCacheMetricsTest > shouldRecordCorrectBlockCacheUsage(RocksDBStore, 
StateStoreContext) > "shouldRecordCorrectBlockCacheUsage(RocksDBStore, 
StateStoreContext).org.apache.kafka.streams.state.internals.RocksDBStore@21450010,
 org.apache.kafka.test.MockInternalProcessorContext@6b9155cf" PASSED

Gradle Test Run :streams:test > Gradle Test Executor 91 > 
RocksDBBlockCacheMetricsTest > shouldRecordCorrectBlockCacheUsage(RocksDBStore, 
StateStoreContext) > "shouldRecordCorrectBlockCacheUsage(RocksDBStore, 
StateStoreContext).org.apache.kafka.streams.state.internals.RocksDBTimestampedStore@8bebb52,
 org.apache.kafka.test.MockInternalProcessorContext@605e33b9" STARTED

Gradle Test Run :streams:test > Gradle Test Executor 91 > 
RocksDBBlockCacheMetricsTest > shouldRecordCorrectBlockCacheUsage(RocksDBStore, 
StateStoreContext) > "shouldRecordCorrectBlockCacheUsage(RocksDBStore, 
StateStoreContext).org.apache.kafka.streams.state.internals.RocksDBTimestampedStore@8bebb52,
 org.apache.kafka.test.MockInternalProcessorContext@605e33b9" PASSED

Gradle Test Run :streams:test > Gradle Test Executor 91 > 
RocksDBBlockCacheMetricsTest > 
shouldRecordCorrectBlockCachePinnedUsage(RocksDBStore, StateStoreContext) > 
"shouldRecordCorrectBlockCachePinnedUsage(RocksDBStore, 
StateStoreContext).org.apache.kafka.streams.state.internals.RocksDBStore@7d0bfb6,
 org.apache.kafka.test.MockInternalProcessorContext@49299746" STARTED

Gradle Test Run :streams:test > Gradle Test Executor 91 > 
RocksDBBlockCacheMetricsTest > 
shouldRecordCorrectBlockCachePinnedUsage(RocksDBStore, StateStoreContext) > 
"shouldRecordCorrectBlockCachePinnedUsage(RocksDBStore, 
StateStoreContext).org.apache.kafka.streams.state.internals.RocksDBStore@7d0bfb6,
 org.apache.kafka.test.MockInternalProcessorContext@49299746" PASSED

Gradle Test Run :streams:test > Gradle Test Executor 91 > 
RocksDBBlockCacheMetricsTest > 
shouldRecordCorrectBlockCachePinnedUsage(RocksDBStore, StateStoreContext) > 
"shouldRecordCorrectBlockCachePinnedUsage(RocksDBStore, 
StateStoreContext).org.apache.kafka.streams.state.internals.RocksDBTimestampedStore@365e374c,
 org.apache.kafka.test.MockInternalProcessorContext@6d908599" STARTED

Gradle Test Run 

Re: [VOTE] KIP-1013: Drop broker and tools support for Java 11 in Kafka 4.0 (deprecate in 3.7)

[David Arthur]

+1 binding

Thanks!
David

On Wed, Jan 3, 2024 at 8:19 PM Ismael Juma  wrote:

> Hi Mickael,
>
> Good catch. I fixed that and one other (similar) case (they were remnants
> of an earlier version of the proposal).
>
> Ismael
>
> On Wed, Jan 3, 2024 at 8:59 AM Mickael Maison 
> wrote:
>
> > Hi Ismael,
> >
> > I'm +1 (binding) too.
> >
> > One small typo, the KIP states "The remaining modules (clients,
> > streams, connect, tools, etc.) will continue to support Java 11.". I
> > think we want to remove support for Java 11 in the tools module so it
> > shouldn't be listed here.
> >
> > Thanks,
> > Mickael
> >
> > On Wed, Jan 3, 2024 at 11:09 AM Divij Vaidya 
> > wrote:
> > >
> > > +1 (binding)
> > >
> > > --
> > > Divij Vaidya
> > >
> > >
> > >
> > > On Wed, Jan 3, 2024 at 11:06 AM Viktor Somogyi-Vass
> > >  wrote:
> > >
> > > > Hi Ismael,
> > > >
> > > > I think it's important to make this change, the youtube video you
> > posted on
> > > > the discussion thread makes very good arguments and so does the KIP.
> > Java 8
> > > > is almost a liability and Java 11 already has smaller (and
> decreasing)
> > > > adoption than 17. It's a +1 (binding) from me.
> > > >
> > > > Thanks,
> > > > Viktor
> > > >
> > > > On Wed, Jan 3, 2024 at 7:00 AM Kamal Chandraprakash <
> > > > kamal.chandraprak...@gmail.com> wrote:
> > > >
> > > > > +1 (non-binding).
> > > > >
> > > > > On Wed, Jan 3, 2024 at 8:01 AM Satish Duggana <
> > satish.dugg...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Thanks Ismael for the proposal.
> > > > > >
> > > > > > Adopting JDK 17 enhances developer productivity and has reached a
> > > > > > level of maturity that has led to its adoption by several other
> > major
> > > > > > projects, signifying its reliability and effectiveness.
> > > > > >
> > > > > > +1 (binding)
> > > > > >
> > > > > >
> > > > > > ~Satish.
> > > > > >
> > > > > > On Wed, 3 Jan 2024 at 06:59, Justine Olshan
> > > > > >  wrote:
> > > > > > >
> > > > > > > Thanks for driving this.
> > > > > > >
> > > > > > > +1 (binding) from me.
> > > > > > >
> > > > > > > Justine
> > > > > > >
> > > > > > > On Tue, Jan 2, 2024 at 4:30 PM Ismael Juma 
> > > > wrote:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > I would like to start a vote on KIP-1013.
> > > > > > > >
> > > > > > > > As stated in the discussion thread, this KIP was proposed
> > after the
> > > > > KIP
> > > > > > > > freeze for Apache Kafka 3.7, but it is purely a documentation
> > > > update
> > > > > > (if we
> > > > > > > > decide to adopt it) and I believe it would serve our users
> > best if
> > > > we
> > > > > > > > communicate the deprecation for removal sooner (i.e. 3.7)
> > rather
> > > > than
> > > > > > later
> > > > > > > > (i.e. 3.8).
> > > > > > > >
> > > > > > > > Please take a look and cast your vote.
> > > > > > > >
> > > > > > > > Link:
> > > > > > > >
> > > > > >
> > > > >
> > > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789510
> > > > > > > >
> > > > > > > > Ismael
> > > > > > > >
> > > > > >
> > > > >
> > > >
> >
>


-- 
David Arthur

Re: [PR] 3.7: Add documentation for Kafka 3.7 [kafka-site]

[via GitHub]

stanislavkozlovski merged PR #576:
URL: https://github.com/apache/kafka-site/pull/576


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] 3.7: Add documentation for Kafka 3.7 [kafka-site]

[via GitHub]

jolshan commented on PR #576:
URL: https://github.com/apache/kafka-site/pull/576#issuecomment-1881984697

   Sorry I was unclear. I meant to ask how to get a diff between versions. 
Looks like Colin did it though.  


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #2549

[Apache Jenkins Server]

See 

Re: [PR] 3.7: Add documentation for Kafka 3.7 [kafka-site]

[via GitHub]

stanislavkozlovski commented on PR #576:
URL: https://github.com/apache/kafka-site/pull/576#issuecomment-1881968438

   @jolshan this is the way afaict, we re-generate everything and keep a copy 
of it under the appropriate `{release}/` folder


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Created] (KAFKA-16095) Update list group state type filter to include the states for the new consumer group type

[Ritika Reddy (Jira)]

Ritika Reddy created KAFKA-16095:


 Summary: Update list group state type filter to include the states 
for the new consumer group type
 Key: KAFKA-16095
 URL: https://issues.apache.org/jira/browse/KAFKA-16095
 Project: Kafka
  Issue Type: Sub-task
Reporter: Ritika Reddy


# While using *—list —state* the current accepted values correspond to the 
classic group type states. We need to include support for the new group type 
states.
 # Consumer Group: Should list the state of the group. Accepted Values: 
 # _UNKNOWN(“unknown”)_
 # {_}EMPTY{_}("empty"),
 # *{_}ASSIGNING{_}("assigning"),*
 # *{_}RECONCILING{_}("reconciling"),*
 # {_}STABLE{_}("stable"),
 # {_}DEAD{_}("dead");


 # Classic Group : Should list the state of the group. Accepted Values: 
 # {_}UNKNOWN{_}("Unknown"),
 # {_}EMPTY{_}("Empty");
 # *{_}PREPARING_REBALANCE{_}("PreparingRebalance"),*
 # *{_}COMPLETING_REBALANCE{_}("CompletingRebalance"),*
 # {_}STABLE{_}("Stable"),
 # {_}DEAD{_}("Dead")



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Apache Kafka 3.7.0 Release

[Colin McCabe]

On an unrelated note, I found a blocker bug related to upgrades from 3.6 (and 
earlier) to 3.7.

The JIRA is here: 
  https://issues.apache.org/jira/browse/KAFKA-16094

Fix here:
  https://github.com/apache/kafka/pull/15153

best,
Colin


On Mon, Jan 8, 2024, at 14:47, Colin McCabe wrote:
> Hi Ismael,
>
> I wasn't aware of that. If we are required to publish all modules, then 
> this is working as intended.
>
> I am a bit curious if we've discussed why we need to publish the server 
> modules to Sonatype. Is there a discussion about the pros and cons of 
> this somewhere?
>
> regards,
> Colin
>
> On Mon, Jan 8, 2024, at 14:09, Ismael Juma wrote:
>> All modules are published to Sonatype - that's a requirement. You may be
>> missing the fact that `core` is published as `kafka_2.13` and `kafka_2.12`.
>>
>> Ismael
>>
>> On Tue, Jan 9, 2024 at 12:00 AM Colin McCabe  wrote:
>>
>>> Hi Ismael,
>>>
>>> It seems like both the metadata gradle module and the server-common module
>>> are getting published to Sonatype as separate artifacts, unless I'm
>>> misunderstanding something. Example:
>>>
>>> https://central.sonatype.com/search?q=kafka-server-common
>>>
>>> I don't see kafka-core getting published, but maybe other private
>>> server-side gradle modules are getting published.
>>>
>>> This seems bad. Is there a reason to publish modules that are only used by
>>> the server on Sonatype?
>>>
>>> best,
>>> Colin
>>>
>>>
>>> On Mon, Jan 8, 2024, at 12:50, Ismael Juma wrote:
>>> > Hi Colin,
>>> >
>>> > I think you may have misunderstood what they mean by gradle metadata -
>>> it's
>>> > not the Kafka metadata module.
>>> >
>>> > Ismael
>>> >
>>> > On Mon, Jan 8, 2024 at 9:45 PM Colin McCabe  wrote:
>>> >
>>> >> Oops, hit send too soon. I see that #15127 was already merged. So we
>>> >> should no longer be publishing :metadata as part of the clients
>>> artifacts,
>>> >> right?
>>> >>
>>> >> thanks,
>>> >> Colin
>>> >>
>>> >>
>>> >> On Mon, Jan 8, 2024, at 11:42, Colin McCabe wrote:
>>> >> > Hi Apporv,
>>> >> >
>>> >> > Please remove the metadata module from any artifacts published for
>>> >> > clients. It is only used by the server.
>>> >> >
>>> >> > best,
>>> >> > Colin
>>> >> >
>>> >> >
>>> >> > On Sun, Jan 7, 2024, at 03:04, Apoorv Mittal wrote:
>>> >> >> Hi Colin,
>>> >> >> Thanks for the response. The only reason for asking the question of
>>> >> >> publishing the metadata is because that's present in previous client
>>> >> >> releases. For more context, the description of PR
>>> >> >>  holds the details and
>>> >> waiting
>>> >> >> for the confirmation there prior to the merge.
>>> >> >>
>>> >> >> Regards,
>>> >> >> Apoorv Mittal
>>> >> >> +44 7721681581
>>> >> >>
>>> >> >>
>>> >> >> On Fri, Jan 5, 2024 at 10:22 PM Colin McCabe 
>>> >> wrote:
>>> >> >>
>>> >> >>> metadata is an internal gradle module. It is not used by clients.
>>> So I
>>> >> >>> don't see why you would want to publish it (unless I'm
>>> misunderstanding
>>> >> >>> something).
>>> >> >>>
>>> >> >>> best,
>>> >> >>> Colin
>>> >> >>>
>>> >> >>>
>>> >> >>> On Fri, Jan 5, 2024, at 10:05, Stanislav Kozlovski wrote:
>>> >> >>> > Thanks for reporting the blockers, folks. Good job finding.
>>> >> >>> >
>>> >> >>> > I have one ask - can anybody with Gradle expertise help review
>>> this
>>> >> small
>>> >> >>> > PR? https://github.com/apache/kafka/pull/15127 (+1, -1)
>>> >> >>> > In particular, we are wondering whether we need to publish module
>>> >> >>> metadata
>>> >> >>> > as part of the gradle publishing process.
>>> >> >>> >
>>> >> >>> >
>>> >> >>> > On Fri, Jan 5, 2024 at 3:56 PM Proven Provenzano
>>> >> >>> >  wrote:
>>> >> >>> >
>>> >> >>> >> We have potentially one more blocker
>>> >> >>> >> https://issues.apache.org/jira/browse/KAFKA-16082 which might
>>> >> cause a
>>> >> >>> data
>>> >> >>> >> loss scenario with JBOD in KRaft.
>>> >> >>> >> Initial analysis thought this is a problem and further review
>>> looks
>>> >> >>> like it
>>> >> >>> >> isn't but we are continuing to dig into the issue to ensure that
>>> it
>>> >> >>> isn't.
>>> >> >>> >> We would request feedback on the bug from anyone who is familiar
>>> >> with
>>> >> >>> this
>>> >> >>> >> code.
>>> >> >>> >>
>>> >> >>> >> --Proven
>>> >> >>> >>
>>> >> >>> >
>>> >> >>> >
>>> >> >>> > --
>>> >> >>> > Best,
>>> >> >>> > Stanislav
>>> >> >>>
>>> >>
>>>

Re: Apache Kafka 3.7.0 Release

[Colin McCabe]

Hi Ismael,

I wasn't aware of that. If we are required to publish all modules, then this is 
working as intended.

I am a bit curious if we've discussed why we need to publish the server modules 
to Sonatype. Is there a discussion about the pros and cons of this somewhere?

regards,
Colin

On Mon, Jan 8, 2024, at 14:09, Ismael Juma wrote:
> All modules are published to Sonatype - that's a requirement. You may be
> missing the fact that `core` is published as `kafka_2.13` and `kafka_2.12`.
>
> Ismael
>
> On Tue, Jan 9, 2024 at 12:00 AM Colin McCabe  wrote:
>
>> Hi Ismael,
>>
>> It seems like both the metadata gradle module and the server-common module
>> are getting published to Sonatype as separate artifacts, unless I'm
>> misunderstanding something. Example:
>>
>> https://central.sonatype.com/search?q=kafka-server-common
>>
>> I don't see kafka-core getting published, but maybe other private
>> server-side gradle modules are getting published.
>>
>> This seems bad. Is there a reason to publish modules that are only used by
>> the server on Sonatype?
>>
>> best,
>> Colin
>>
>>
>> On Mon, Jan 8, 2024, at 12:50, Ismael Juma wrote:
>> > Hi Colin,
>> >
>> > I think you may have misunderstood what they mean by gradle metadata -
>> it's
>> > not the Kafka metadata module.
>> >
>> > Ismael
>> >
>> > On Mon, Jan 8, 2024 at 9:45 PM Colin McCabe  wrote:
>> >
>> >> Oops, hit send too soon. I see that #15127 was already merged. So we
>> >> should no longer be publishing :metadata as part of the clients
>> artifacts,
>> >> right?
>> >>
>> >> thanks,
>> >> Colin
>> >>
>> >>
>> >> On Mon, Jan 8, 2024, at 11:42, Colin McCabe wrote:
>> >> > Hi Apporv,
>> >> >
>> >> > Please remove the metadata module from any artifacts published for
>> >> > clients. It is only used by the server.
>> >> >
>> >> > best,
>> >> > Colin
>> >> >
>> >> >
>> >> > On Sun, Jan 7, 2024, at 03:04, Apoorv Mittal wrote:
>> >> >> Hi Colin,
>> >> >> Thanks for the response. The only reason for asking the question of
>> >> >> publishing the metadata is because that's present in previous client
>> >> >> releases. For more context, the description of PR
>> >> >>  holds the details and
>> >> waiting
>> >> >> for the confirmation there prior to the merge.
>> >> >>
>> >> >> Regards,
>> >> >> Apoorv Mittal
>> >> >> +44 7721681581
>> >> >>
>> >> >>
>> >> >> On Fri, Jan 5, 2024 at 10:22 PM Colin McCabe 
>> >> wrote:
>> >> >>
>> >> >>> metadata is an internal gradle module. It is not used by clients.
>> So I
>> >> >>> don't see why you would want to publish it (unless I'm
>> misunderstanding
>> >> >>> something).
>> >> >>>
>> >> >>> best,
>> >> >>> Colin
>> >> >>>
>> >> >>>
>> >> >>> On Fri, Jan 5, 2024, at 10:05, Stanislav Kozlovski wrote:
>> >> >>> > Thanks for reporting the blockers, folks. Good job finding.
>> >> >>> >
>> >> >>> > I have one ask - can anybody with Gradle expertise help review
>> this
>> >> small
>> >> >>> > PR? https://github.com/apache/kafka/pull/15127 (+1, -1)
>> >> >>> > In particular, we are wondering whether we need to publish module
>> >> >>> metadata
>> >> >>> > as part of the gradle publishing process.
>> >> >>> >
>> >> >>> >
>> >> >>> > On Fri, Jan 5, 2024 at 3:56 PM Proven Provenzano
>> >> >>> >  wrote:
>> >> >>> >
>> >> >>> >> We have potentially one more blocker
>> >> >>> >> https://issues.apache.org/jira/browse/KAFKA-16082 which might
>> >> cause a
>> >> >>> data
>> >> >>> >> loss scenario with JBOD in KRaft.
>> >> >>> >> Initial analysis thought this is a problem and further review
>> looks
>> >> >>> like it
>> >> >>> >> isn't but we are continuing to dig into the issue to ensure that
>> it
>> >> >>> isn't.
>> >> >>> >> We would request feedback on the bug from anyone who is familiar
>> >> with
>> >> >>> this
>> >> >>> >> code.
>> >> >>> >>
>> >> >>> >> --Proven
>> >> >>> >>
>> >> >>> >
>> >> >>> >
>> >> >>> > --
>> >> >>> > Best,
>> >> >>> > Stanislav
>> >> >>>
>> >>
>>

[jira] [Created] (KAFKA-16094) 3.7 brokers must be able to register with 3.6 and earlier controllers

[Colin McCabe (Jira)]

Colin McCabe created KAFKA-16094:


 Summary: 3.7 brokers must be able to register with 3.6 and earlier 
controllers
 Key: KAFKA-16094
 URL: https://issues.apache.org/jira/browse/KAFKA-16094
 Project: Kafka
  Issue Type: Bug
Affects Versions: 3.7.0
Reporter: Colin McCabe
Assignee: Colin McCabe


3.7 brokers must be able to register with 3.6 and earlier controllers. So this 
means that the logDirs field must be ignorable (aka, not sent) if the highest 
BrokerRegistrationRequest version we can negotiate is older than v2.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-1014: Managing Unstable Metadata Versions in Apache Kafka

[Justine Olshan]

Hey Colin,

I had some offline discussions with Proven previously and it seems like he
said something different so I'm glad I brought it up here.

Let's clarify if we are ok with reordering unstable metadata versions :)

Justine

On Mon, Jan 8, 2024 at 1:56 PM Colin McCabe  wrote:

> On Mon, Jan 8, 2024, at 13:19, Justine Olshan wrote:
> > Hey all,
> >
> > I was wondering how often we plan to update LATEST_PRODUCTION metadata
> > version. Is this something we should do as soon as the feature is
> complete
> > or something we do when we are releasing kafka. When is the time we
> abandon
> > a MV so that other features can be unblocked?
>
> Hi Justine,
>
> Thanks for reviewing.
>
> The idea is that you should bump LATEST_PRODUCTION when you want to take a
> feature to production. That could mean deploying it internally somewhere to
> production, or doing an Apache release that lets everyone deploy the thing
> to production.
>
> Not in production? No need to care about this. Make any changes you like.
>
> As a corollary, we should keep the LATEST_PRODUCTION version as low as it
> can be. If you haven't tested the feature, don't freeze it in stone yet.
>
> >
> > I am just considering a feature that may end up missing a release. It
> seems
> > like maybe that MV would block future metadata versions until we decide
> the
> > feature won't make the cut. From that point, all "ready" features should
> be
> > able to be released.
>
> The intention is the opposite. A feature in an unstable metadata version
> doesn't block anything. You can always move a feature from one unstable
> metadata version to another if the feature starts taking too long to finish.
>
> > I'm also wondering if the KIP should include some information about how a
> > metadata should be abandoned. Maybe there is a specific message to write
> in
> > the file? So folks who were maybe waiting on that version know they can
> > release their feature?
> >
> > I am also assuming that we don't shift all the waiting metadata versions
> > when we abandon a version, but it would be good to clarify and include in
> > the KIP.
>
> I'm not sure what you mean by abandoning a version. We never abandon a
> version once it's stable.
>
> Unstable versions can change. I wouldn't describe this as "abandonment",
> just the MV changing prior to release.
>
> In a similar way, the contents of the 3.7 branch will change up until
> 3.7.0 is released. Once it gets released, it's never unreleased. We just
> move on to 3.7.1. Same thing here.
>
> best,
> Colin
>
> >
> > Thanks,
> >
> > Justine
> >
> > On Mon, Jan 8, 2024 at 12:44 PM Colin McCabe  wrote:
> >
> >> Hi Proven,
> >>
> >> Thanks for the KIP. I think there is a need for this capability, for
> those
> >> of us who deploy from trunk (or branches dervied from trunk).
> >>
> >> With regard to "unstable.metadata.versions.enable": is this going to be
> a
> >> documented configuration, or an internal one? I am guessing we want it
> to
> >> be documented, so that users can use it. If we do, we should probably
> also
> >> very prominently warn that THIS WILL BREAK UPGRADES FOR YOUR CLUSTER.
> That
> >> includes logging an ERROR message on startup, etc.
> >>
> >> It would be good to document if a release can go out that contains
> "future
> >> MVs" that are unstable. Like can we make a 3.8 release that contains
> >> IBP_4_0_IV0 in MetadataVersion.java, as an unstable future MV?
> Personally I
> >> think the answer should be "yes," but with the usual caveats. When the
> >> actual 4.0 comes out, the unstable 4.0 MV that shipped in 3.8 probably
> >> won't work, and you won't be able to upgrade. (It was unstable, we told
> you
> >> not to use it.)
> >>
> >> best,
> >> Colin
> >>
> >>
> >> On Fri, Jan 5, 2024, at 07:32, Proven Provenzano wrote:
> >> > Hey folks,
> >> >
> >> > I am starting a discussion thread for managing unstable metadata
> >> > versions
> >> > in Apache Kafka.
> >> >
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1014%3A+Managing+Unstable+Metadata+Versions+in+Apache+Kafka
> >> >
> >> > This KIP is actually already implemented in 3.7 with PR
> >> > https://github.com/apache/kafka/pull/14860.
> >> > I have created this KIP to explain the motivation and how managing
> >> Metadata
> >> > Versions is expected to work.
> >> > Comments are greatly appreciated as this process can always be
> improved.
> >> >
> >> > --
> >> > --Proven
> >>
>

Re: Apache Kafka 3.7.0 Release

[Ismael Juma]

All modules are published to Sonatype - that's a requirement. You may be
missing the fact that `core` is published as `kafka_2.13` and `kafka_2.12`.

Ismael

On Tue, Jan 9, 2024 at 12:00 AM Colin McCabe  wrote:

> Hi Ismael,
>
> It seems like both the metadata gradle module and the server-common module
> are getting published to Sonatype as separate artifacts, unless I'm
> misunderstanding something. Example:
>
> https://central.sonatype.com/search?q=kafka-server-common
>
> I don't see kafka-core getting published, but maybe other private
> server-side gradle modules are getting published.
>
> This seems bad. Is there a reason to publish modules that are only used by
> the server on Sonatype?
>
> best,
> Colin
>
>
> On Mon, Jan 8, 2024, at 12:50, Ismael Juma wrote:
> > Hi Colin,
> >
> > I think you may have misunderstood what they mean by gradle metadata -
> it's
> > not the Kafka metadata module.
> >
> > Ismael
> >
> > On Mon, Jan 8, 2024 at 9:45 PM Colin McCabe  wrote:
> >
> >> Oops, hit send too soon. I see that #15127 was already merged. So we
> >> should no longer be publishing :metadata as part of the clients
> artifacts,
> >> right?
> >>
> >> thanks,
> >> Colin
> >>
> >>
> >> On Mon, Jan 8, 2024, at 11:42, Colin McCabe wrote:
> >> > Hi Apporv,
> >> >
> >> > Please remove the metadata module from any artifacts published for
> >> > clients. It is only used by the server.
> >> >
> >> > best,
> >> > Colin
> >> >
> >> >
> >> > On Sun, Jan 7, 2024, at 03:04, Apoorv Mittal wrote:
> >> >> Hi Colin,
> >> >> Thanks for the response. The only reason for asking the question of
> >> >> publishing the metadata is because that's present in previous client
> >> >> releases. For more context, the description of PR
> >> >>  holds the details and
> >> waiting
> >> >> for the confirmation there prior to the merge.
> >> >>
> >> >> Regards,
> >> >> Apoorv Mittal
> >> >> +44 7721681581
> >> >>
> >> >>
> >> >> On Fri, Jan 5, 2024 at 10:22 PM Colin McCabe 
> >> wrote:
> >> >>
> >> >>> metadata is an internal gradle module. It is not used by clients.
> So I
> >> >>> don't see why you would want to publish it (unless I'm
> misunderstanding
> >> >>> something).
> >> >>>
> >> >>> best,
> >> >>> Colin
> >> >>>
> >> >>>
> >> >>> On Fri, Jan 5, 2024, at 10:05, Stanislav Kozlovski wrote:
> >> >>> > Thanks for reporting the blockers, folks. Good job finding.
> >> >>> >
> >> >>> > I have one ask - can anybody with Gradle expertise help review
> this
> >> small
> >> >>> > PR? https://github.com/apache/kafka/pull/15127 (+1, -1)
> >> >>> > In particular, we are wondering whether we need to publish module
> >> >>> metadata
> >> >>> > as part of the gradle publishing process.
> >> >>> >
> >> >>> >
> >> >>> > On Fri, Jan 5, 2024 at 3:56 PM Proven Provenzano
> >> >>> >  wrote:
> >> >>> >
> >> >>> >> We have potentially one more blocker
> >> >>> >> https://issues.apache.org/jira/browse/KAFKA-16082 which might
> >> cause a
> >> >>> data
> >> >>> >> loss scenario with JBOD in KRaft.
> >> >>> >> Initial analysis thought this is a problem and further review
> looks
> >> >>> like it
> >> >>> >> isn't but we are continuing to dig into the issue to ensure that
> it
> >> >>> isn't.
> >> >>> >> We would request feedback on the bug from anyone who is familiar
> >> with
> >> >>> this
> >> >>> >> code.
> >> >>> >>
> >> >>> >> --Proven
> >> >>> >>
> >> >>> >
> >> >>> >
> >> >>> > --
> >> >>> > Best,
> >> >>> > Stanislav
> >> >>>
> >>
>

Re: [VOTE] KIP-1012: The need for a Kafka 3.8.x release

[Colin McCabe]

Hi Chris,

Based on some of the offline discussions we had, people generally feel that 
major/minor releases have a large enough overhead that they don't want them 
more frequently than every 4 months. (Obviously dot releases are a different 
story) So Josep and I didn't want to raise the issue here.

I also feel that 4 months should be enough (for anyone? :) ) But I'm always an 
optimist.

best,
Colin


On Mon, Jan 8, 2024, at 13:03, Chris Egerton wrote:
> Hi Colin,
>
> The idea isn't to hold up 4.0.0 any longer; in fact, it's the opposite. If
> things slip a bit with 3.8.0 and we take, for example, an extra month to
> deliver it (or even to cut the branch), I don't think this should
> necessarily translate to an extra month of latency between now and 4.0.0,
> given exactly what you mention about the major changes we plan to include
> in 4.0.0 (which consist more of dropping support for existing things than
> adding support for new things).
>
> If we want to avoid confusion, we could say something like "no later than 3
> to 4 months after the 3.8 branch is created". Frankly though, I think it's
> unnecessary to specify an exact timeline for 4.0 in this KIP, since nothing
> in the proposal actually diverges from the usual time-based release plan we
> follow. The only necessary part seems to be to state that 4.0 will directly
> follow 3.8 (as opposed to 3.9, 3.10, etc.). But perhaps I'm missing
> something?
>
> Cheers,
>
> Chris
>
> On Mon, Jan 8, 2024 at 2:38 PM Colin McCabe  wrote:
>
>> On Mon, Jan 8, 2024, at 09:05, Chris Egerton wrote:
>> > Hi Josep,
>> >
>> > Thanks for the KIP! +1 (binding).
>> >
>> > One small nit: I don't think we necessarily have to hold ourselves to
>> > releasing 4.0.0 "3 to 4 months after 3.8 branch is created" (quoting the
>> > timeline section of the KIP). IMO it's fine to leave some wiggle room for
>> > the 4.0.0 release without codifying a timeline in this KIP. Maybe
>> something
>> > like "some time after 3.8 branch is created" would be sufficient?
>> Anyways,
>> > not a huge thing, I'm sure we'll all give 4.0.0 the flexibility it needs
>> > with the understanding that this KIP is more focused on 3.8.0 than 4.0.0.
>> >
>>
>> Hmm... I don't see any obstacles in the path of releasing 4.0 after the
>> traditional 4 months of development. Keep in mind, we're removing things
>> from the code (the ability to support JDK8, ZooKeeper mode, etc.), not
>> adding things. We already support JDK11 so saying that it's the minimum is
>> a very quick change. Similarly, we already support KRaft so saying that
>> it's the only mode should be a pretty quick change.
>>
>> Also, we added a note that "the timeline is very rough" to KIP-833 and it
>> caued all kinds of confusion. So overall I'd prefer to leave the language
>> about 4.0 unchanged.
>>
>> best,
>> Colin
>>
>> >
>> > Cheers,
>> >
>> > Chris
>> >
>> > On Mon, Jan 8, 2024 at 11:41 AM Greg Harris > >
>> > wrote:
>> >
>> >> Thanks Josep for leading the KIP and building consensus on 3.8!
>> >>
>> >> +1 (binding)
>> >>
>> >> Greg
>> >>
>> >> On Sun, Jan 7, 2024 at 11:45 PM Josep Prat > >
>> >> wrote:
>> >> >
>> >> > Hi all,
>> >> >
>> >> > Thanks for your comments,
>> >> > I reworded some parts of the KIP to express that:
>> >> > - The KIP is to agree that we need at least one more minor version in
>> the
>> >> > 3.x series
>> >> > - Explicitly saying that the list of KIPs is not exhaustive and that
>> if
>> >> > some are not done, we would need another minor version
>> >> > - Which are the KIPs/Features the community identified that should be
>> >> > present in a 3.x version so they can safely migrate to a potential 4.0
>> >> > version
>> >> > - The timeline of 4.0 (starting after branching, not after release)
>> >> > - Wording is now focusing more on the need to have at least another
>> minor
>> >> > release in 3.x to enable and ease migration to a potential 4.0 version
>> >> >
>> >> > I always mention potential in terms of 4.0 as we don't know what will
>> be
>> >> in
>> >> > there yet, and this KIP's scope is not meant to define this.
>> >> >
>> >> > Best,
>> >> >
>> >> > On Fri, Jan 5, 2024 at 10:46 PM Ismael Juma 
>> wrote:
>> >> >
>> >> > > I agree with Colin. Also, 4.0 would be started after 3.8 is branched
>> >> (not
>> >> > > after 3.8.0 is released). The rest looks good.
>> >> > >
>> >> > > Thanks!
>> >> > >
>> >> > > Ismael
>> >> > >
>> >> > > On Fri, Jan 5, 2024 at 11:27 PM Colin McCabe 
>> >> wrote:
>> >> > >
>> >> > > > Hi,
>> >> > > >
>> >> > > > Thanks for calling the vote, Josep.
>> >> > > >
>> >> > > > I re-checked this, and saw something that we missed updating. One
>> >> thing
>> >> > > we
>> >> > > > talked about earlier is that KIP-966 is actually not required for
>> >> 3.8.
>> >> > > What
>> >> > > > is required is some way of enabling unclean leader election by
>> >> default in
>> >> > > > KRaft. (Which could be KIP-966, or something else). Please revise
>> >> this.
>> >> > > >
>> >> > > > best,
>> >> > > > Colin

Re: Apache Kafka 3.7.0 Release

[Colin McCabe]

Hi Ismael,

It seems like both the metadata gradle module and the server-common module are 
getting published to Sonatype as separate artifacts, unless I'm 
misunderstanding something. Example:

https://central.sonatype.com/search?q=kafka-server-common

I don't see kafka-core getting published, but maybe other private server-side 
gradle modules are getting published.

This seems bad. Is there a reason to publish modules that are only used by the 
server on Sonatype?

best,
Colin


On Mon, Jan 8, 2024, at 12:50, Ismael Juma wrote:
> Hi Colin,
>
> I think you may have misunderstood what they mean by gradle metadata - it's
> not the Kafka metadata module.
>
> Ismael
>
> On Mon, Jan 8, 2024 at 9:45 PM Colin McCabe  wrote:
>
>> Oops, hit send too soon. I see that #15127 was already merged. So we
>> should no longer be publishing :metadata as part of the clients artifacts,
>> right?
>>
>> thanks,
>> Colin
>>
>>
>> On Mon, Jan 8, 2024, at 11:42, Colin McCabe wrote:
>> > Hi Apporv,
>> >
>> > Please remove the metadata module from any artifacts published for
>> > clients. It is only used by the server.
>> >
>> > best,
>> > Colin
>> >
>> >
>> > On Sun, Jan 7, 2024, at 03:04, Apoorv Mittal wrote:
>> >> Hi Colin,
>> >> Thanks for the response. The only reason for asking the question of
>> >> publishing the metadata is because that's present in previous client
>> >> releases. For more context, the description of PR
>> >>  holds the details and
>> waiting
>> >> for the confirmation there prior to the merge.
>> >>
>> >> Regards,
>> >> Apoorv Mittal
>> >> +44 7721681581
>> >>
>> >>
>> >> On Fri, Jan 5, 2024 at 10:22 PM Colin McCabe 
>> wrote:
>> >>
>> >>> metadata is an internal gradle module. It is not used by clients. So I
>> >>> don't see why you would want to publish it (unless I'm misunderstanding
>> >>> something).
>> >>>
>> >>> best,
>> >>> Colin
>> >>>
>> >>>
>> >>> On Fri, Jan 5, 2024, at 10:05, Stanislav Kozlovski wrote:
>> >>> > Thanks for reporting the blockers, folks. Good job finding.
>> >>> >
>> >>> > I have one ask - can anybody with Gradle expertise help review this
>> small
>> >>> > PR? https://github.com/apache/kafka/pull/15127 (+1, -1)
>> >>> > In particular, we are wondering whether we need to publish module
>> >>> metadata
>> >>> > as part of the gradle publishing process.
>> >>> >
>> >>> >
>> >>> > On Fri, Jan 5, 2024 at 3:56 PM Proven Provenzano
>> >>> >  wrote:
>> >>> >
>> >>> >> We have potentially one more blocker
>> >>> >> https://issues.apache.org/jira/browse/KAFKA-16082 which might
>> cause a
>> >>> data
>> >>> >> loss scenario with JBOD in KRaft.
>> >>> >> Initial analysis thought this is a problem and further review looks
>> >>> like it
>> >>> >> isn't but we are continuing to dig into the issue to ensure that it
>> >>> isn't.
>> >>> >> We would request feedback on the bug from anyone who is familiar
>> with
>> >>> this
>> >>> >> code.
>> >>> >>
>> >>> >> --Proven
>> >>> >>
>> >>> >
>> >>> >
>> >>> > --
>> >>> > Best,
>> >>> > Stanislav
>> >>>
>>

Re: [DISCUSS] KIP-1014: Managing Unstable Metadata Versions in Apache Kafka

[Colin McCabe]

On Mon, Jan 8, 2024, at 13:19, Justine Olshan wrote:
> Hey all,
>
> I was wondering how often we plan to update LATEST_PRODUCTION metadata
> version. Is this something we should do as soon as the feature is complete
> or something we do when we are releasing kafka. When is the time we abandon
> a MV so that other features can be unblocked?

Hi Justine,

Thanks for reviewing.

The idea is that you should bump LATEST_PRODUCTION when you want to take a 
feature to production. That could mean deploying it internally somewhere to 
production, or doing an Apache release that lets everyone deploy the thing to 
production.

Not in production? No need to care about this. Make any changes you like.

As a corollary, we should keep the LATEST_PRODUCTION version as low as it can 
be. If you haven't tested the feature, don't freeze it in stone yet.

>
> I am just considering a feature that may end up missing a release. It seems
> like maybe that MV would block future metadata versions until we decide the
> feature won't make the cut. From that point, all "ready" features should be
> able to be released.

The intention is the opposite. A feature in an unstable metadata version 
doesn't block anything. You can always move a feature from one unstable 
metadata version to another if the feature starts taking too long to finish.

> I'm also wondering if the KIP should include some information about how a
> metadata should be abandoned. Maybe there is a specific message to write in
> the file? So folks who were maybe waiting on that version know they can
> release their feature?
>
> I am also assuming that we don't shift all the waiting metadata versions
> when we abandon a version, but it would be good to clarify and include in
> the KIP.

I'm not sure what you mean by abandoning a version. We never abandon a version 
once it's stable.

Unstable versions can change. I wouldn't describe this as "abandonment", just 
the MV changing prior to release.

In a similar way, the contents of the 3.7 branch will change up until 3.7.0 is 
released. Once it gets released, it's never unreleased. We just move on to 
3.7.1. Same thing here.

best,
Colin

>
> Thanks,
>
> Justine
>
> On Mon, Jan 8, 2024 at 12:44 PM Colin McCabe  wrote:
>
>> Hi Proven,
>>
>> Thanks for the KIP. I think there is a need for this capability, for those
>> of us who deploy from trunk (or branches dervied from trunk).
>>
>> With regard to "unstable.metadata.versions.enable": is this going to be a
>> documented configuration, or an internal one? I am guessing we want it to
>> be documented, so that users can use it. If we do, we should probably also
>> very prominently warn that THIS WILL BREAK UPGRADES FOR YOUR CLUSTER. That
>> includes logging an ERROR message on startup, etc.
>>
>> It would be good to document if a release can go out that contains "future
>> MVs" that are unstable. Like can we make a 3.8 release that contains
>> IBP_4_0_IV0 in MetadataVersion.java, as an unstable future MV? Personally I
>> think the answer should be "yes," but with the usual caveats. When the
>> actual 4.0 comes out, the unstable 4.0 MV that shipped in 3.8 probably
>> won't work, and you won't be able to upgrade. (It was unstable, we told you
>> not to use it.)
>>
>> best,
>> Colin
>>
>>
>> On Fri, Jan 5, 2024, at 07:32, Proven Provenzano wrote:
>> > Hey folks,
>> >
>> > I am starting a discussion thread for managing unstable metadata
>> > versions
>> > in Apache Kafka.
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1014%3A+Managing+Unstable+Metadata+Versions+in+Apache+Kafka
>> >
>> > This KIP is actually already implemented in 3.7 with PR
>> > https://github.com/apache/kafka/pull/14860.
>> > I have created this KIP to explain the motivation and how managing
>> Metadata
>> > Versions is expected to work.
>> > Comments are greatly appreciated as this process can always be improved.
>> >
>> > --
>> > --Proven
>>

Re: [DISCUSS] KIP-1014: Managing Unstable Metadata Versions in Apache Kafka

[Justine Olshan]

Hey all,

I was wondering how often we plan to update LATEST_PRODUCTION metadata
version. Is this something we should do as soon as the feature is complete
or something we do when we are releasing kafka. When is the time we abandon
a MV so that other features can be unblocked?

I am just considering a feature that may end up missing a release. It seems
like maybe that MV would block future metadata versions until we decide the
feature won't make the cut. From that point, all "ready" features should be
able to be released.
I'm also wondering if the KIP should include some information about how a
metadata should be abandoned. Maybe there is a specific message to write in
the file? So folks who were maybe waiting on that version know they can
release their feature?

I am also assuming that we don't shift all the waiting metadata versions
when we abandon a version, but it would be good to clarify and include in
the KIP.

Thanks,

Justine

On Mon, Jan 8, 2024 at 12:44 PM Colin McCabe  wrote:

> Hi Proven,
>
> Thanks for the KIP. I think there is a need for this capability, for those
> of us who deploy from trunk (or branches dervied from trunk).
>
> With regard to "unstable.metadata.versions.enable": is this going to be a
> documented configuration, or an internal one? I am guessing we want it to
> be documented, so that users can use it. If we do, we should probably also
> very prominently warn that THIS WILL BREAK UPGRADES FOR YOUR CLUSTER. That
> includes logging an ERROR message on startup, etc.
>
> It would be good to document if a release can go out that contains "future
> MVs" that are unstable. Like can we make a 3.8 release that contains
> IBP_4_0_IV0 in MetadataVersion.java, as an unstable future MV? Personally I
> think the answer should be "yes," but with the usual caveats. When the
> actual 4.0 comes out, the unstable 4.0 MV that shipped in 3.8 probably
> won't work, and you won't be able to upgrade. (It was unstable, we told you
> not to use it.)
>
> best,
> Colin
>
>
> On Fri, Jan 5, 2024, at 07:32, Proven Provenzano wrote:
> > Hey folks,
> >
> > I am starting a discussion thread for managing unstable metadata
> > versions
> > in Apache Kafka.
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1014%3A+Managing+Unstable+Metadata+Versions+in+Apache+Kafka
> >
> > This KIP is actually already implemented in 3.7 with PR
> > https://github.com/apache/kafka/pull/14860.
> > I have created this KIP to explain the motivation and how managing
> Metadata
> > Versions is expected to work.
> > Comments are greatly appreciated as this process can always be improved.
> >
> > --
> > --Proven
>

Re: Apache Kafka 3.7.0 Release

[David Jacot]

Hi all,

Are you talking about publishing the artefacts to maven central? Looking at
the history [1], it seems that the metadata module has been published since
we have it. I also see other internal modules there too.

[1]
https://central.sonatype.com/artifact/org.apache.kafka/kafka-metadata/versions

Best,
David

Le lun. 8 janv. 2024 à 21:51, Ismael Juma  a écrit :

> Hi Colin,
>
> I think you may have misunderstood what they mean by gradle metadata - it's
> not the Kafka metadata module.
>
> Ismael
>
> On Mon, Jan 8, 2024 at 9:45 PM Colin McCabe  wrote:
>
> > Oops, hit send too soon. I see that #15127 was already merged. So we
> > should no longer be publishing :metadata as part of the clients
> artifacts,
> > right?
> >
> > thanks,
> > Colin
> >
> >
> > On Mon, Jan 8, 2024, at 11:42, Colin McCabe wrote:
> > > Hi Apporv,
> > >
> > > Please remove the metadata module from any artifacts published for
> > > clients. It is only used by the server.
> > >
> > > best,
> > > Colin
> > >
> > >
> > > On Sun, Jan 7, 2024, at 03:04, Apoorv Mittal wrote:
> > >> Hi Colin,
> > >> Thanks for the response. The only reason for asking the question of
> > >> publishing the metadata is because that's present in previous client
> > >> releases. For more context, the description of PR
> > >>  holds the details and
> > waiting
> > >> for the confirmation there prior to the merge.
> > >>
> > >> Regards,
> > >> Apoorv Mittal
> > >> +44 7721681581
> > >>
> > >>
> > >> On Fri, Jan 5, 2024 at 10:22 PM Colin McCabe 
> > wrote:
> > >>
> > >>> metadata is an internal gradle module. It is not used by clients. So
> I
> > >>> don't see why you would want to publish it (unless I'm
> misunderstanding
> > >>> something).
> > >>>
> > >>> best,
> > >>> Colin
> > >>>
> > >>>
> > >>> On Fri, Jan 5, 2024, at 10:05, Stanislav Kozlovski wrote:
> > >>> > Thanks for reporting the blockers, folks. Good job finding.
> > >>> >
> > >>> > I have one ask - can anybody with Gradle expertise help review this
> > small
> > >>> > PR? https://github.com/apache/kafka/pull/15127 (+1, -1)
> > >>> > In particular, we are wondering whether we need to publish module
> > >>> metadata
> > >>> > as part of the gradle publishing process.
> > >>> >
> > >>> >
> > >>> > On Fri, Jan 5, 2024 at 3:56 PM Proven Provenzano
> > >>> >  wrote:
> > >>> >
> > >>> >> We have potentially one more blocker
> > >>> >> https://issues.apache.org/jira/browse/KAFKA-16082 which might
> > cause a
> > >>> data
> > >>> >> loss scenario with JBOD in KRaft.
> > >>> >> Initial analysis thought this is a problem and further review
> looks
> > >>> like it
> > >>> >> isn't but we are continuing to dig into the issue to ensure that
> it
> > >>> isn't.
> > >>> >> We would request feedback on the bug from anyone who is familiar
> > with
> > >>> this
> > >>> >> code.
> > >>> >>
> > >>> >> --Proven
> > >>> >>
> > >>> >
> > >>> >
> > >>> > --
> > >>> > Best,
> > >>> > Stanislav
> > >>>
> >
>

Re: [VOTE] KIP-1012: The need for a Kafka 3.8.x release

[Chris Egerton]

Hi Colin,

The idea isn't to hold up 4.0.0 any longer; in fact, it's the opposite. If
things slip a bit with 3.8.0 and we take, for example, an extra month to
deliver it (or even to cut the branch), I don't think this should
necessarily translate to an extra month of latency between now and 4.0.0,
given exactly what you mention about the major changes we plan to include
in 4.0.0 (which consist more of dropping support for existing things than
adding support for new things).

If we want to avoid confusion, we could say something like "no later than 3
to 4 months after the 3.8 branch is created". Frankly though, I think it's
unnecessary to specify an exact timeline for 4.0 in this KIP, since nothing
in the proposal actually diverges from the usual time-based release plan we
follow. The only necessary part seems to be to state that 4.0 will directly
follow 3.8 (as opposed to 3.9, 3.10, etc.). But perhaps I'm missing
something?

Cheers,

Chris

On Mon, Jan 8, 2024 at 2:38 PM Colin McCabe  wrote:

> On Mon, Jan 8, 2024, at 09:05, Chris Egerton wrote:
> > Hi Josep,
> >
> > Thanks for the KIP! +1 (binding).
> >
> > One small nit: I don't think we necessarily have to hold ourselves to
> > releasing 4.0.0 "3 to 4 months after 3.8 branch is created" (quoting the
> > timeline section of the KIP). IMO it's fine to leave some wiggle room for
> > the 4.0.0 release without codifying a timeline in this KIP. Maybe
> something
> > like "some time after 3.8 branch is created" would be sufficient?
> Anyways,
> > not a huge thing, I'm sure we'll all give 4.0.0 the flexibility it needs
> > with the understanding that this KIP is more focused on 3.8.0 than 4.0.0.
> >
>
> Hmm... I don't see any obstacles in the path of releasing 4.0 after the
> traditional 4 months of development. Keep in mind, we're removing things
> from the code (the ability to support JDK8, ZooKeeper mode, etc.), not
> adding things. We already support JDK11 so saying that it's the minimum is
> a very quick change. Similarly, we already support KRaft so saying that
> it's the only mode should be a pretty quick change.
>
> Also, we added a note that "the timeline is very rough" to KIP-833 and it
> caued all kinds of confusion. So overall I'd prefer to leave the language
> about 4.0 unchanged.
>
> best,
> Colin
>
> >
> > Cheers,
> >
> > Chris
> >
> > On Mon, Jan 8, 2024 at 11:41 AM Greg Harris  >
> > wrote:
> >
> >> Thanks Josep for leading the KIP and building consensus on 3.8!
> >>
> >> +1 (binding)
> >>
> >> Greg
> >>
> >> On Sun, Jan 7, 2024 at 11:45 PM Josep Prat  >
> >> wrote:
> >> >
> >> > Hi all,
> >> >
> >> > Thanks for your comments,
> >> > I reworded some parts of the KIP to express that:
> >> > - The KIP is to agree that we need at least one more minor version in
> the
> >> > 3.x series
> >> > - Explicitly saying that the list of KIPs is not exhaustive and that
> if
> >> > some are not done, we would need another minor version
> >> > - Which are the KIPs/Features the community identified that should be
> >> > present in a 3.x version so they can safely migrate to a potential 4.0
> >> > version
> >> > - The timeline of 4.0 (starting after branching, not after release)
> >> > - Wording is now focusing more on the need to have at least another
> minor
> >> > release in 3.x to enable and ease migration to a potential 4.0 version
> >> >
> >> > I always mention potential in terms of 4.0 as we don't know what will
> be
> >> in
> >> > there yet, and this KIP's scope is not meant to define this.
> >> >
> >> > Best,
> >> >
> >> > On Fri, Jan 5, 2024 at 10:46 PM Ismael Juma 
> wrote:
> >> >
> >> > > I agree with Colin. Also, 4.0 would be started after 3.8 is branched
> >> (not
> >> > > after 3.8.0 is released). The rest looks good.
> >> > >
> >> > > Thanks!
> >> > >
> >> > > Ismael
> >> > >
> >> > > On Fri, Jan 5, 2024 at 11:27 PM Colin McCabe 
> >> wrote:
> >> > >
> >> > > > Hi,
> >> > > >
> >> > > > Thanks for calling the vote, Josep.
> >> > > >
> >> > > > I re-checked this, and saw something that we missed updating. One
> >> thing
> >> > > we
> >> > > > talked about earlier is that KIP-966 is actually not required for
> >> 3.8.
> >> > > What
> >> > > > is required is some way of enabling unclean leader election by
> >> default in
> >> > > > KRaft. (Which could be KIP-966, or something else). Please revise
> >> this.
> >> > > >
> >> > > > best,
> >> > > > Colin
> >> > > >
> >> > > > On Fri, Jan 5, 2024, at 02:50, Anton Agestam wrote:
> >> > > > > +1 from me.
> >> > > > >
> >> > > > > Den fre 5 jan. 2024 kl 10:33 skrev Josep Prat
> >> > > > :
> >> > > > >
> >> > > > >> Hi all,
> >> > > > >>
> >> > > > >> I'd like to start a vote on KIP-1012:
> >> > > > >>
> >> > > > >>
> >> > > >
> >> > >
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1012%3A+The+need+for+a+Kafka+3.8.x+release
> >> > > > >>
> >> > > > >> Discussion thread is here:
> >> > > > >>
> https://lists.apache.org/thread/kvdp2gmq5gd9txkvxh5vk3z2n55b04s5
> >> > > > >>
> >> > > > >> 

Re: Apache Kafka 3.7.0 Release

[Ismael Juma]

Hi Colin,

I think you may have misunderstood what they mean by gradle metadata - it's
not the Kafka metadata module.

Ismael

On Mon, Jan 8, 2024 at 9:45 PM Colin McCabe  wrote:

> Oops, hit send too soon. I see that #15127 was already merged. So we
> should no longer be publishing :metadata as part of the clients artifacts,
> right?
>
> thanks,
> Colin
>
>
> On Mon, Jan 8, 2024, at 11:42, Colin McCabe wrote:
> > Hi Apporv,
> >
> > Please remove the metadata module from any artifacts published for
> > clients. It is only used by the server.
> >
> > best,
> > Colin
> >
> >
> > On Sun, Jan 7, 2024, at 03:04, Apoorv Mittal wrote:
> >> Hi Colin,
> >> Thanks for the response. The only reason for asking the question of
> >> publishing the metadata is because that's present in previous client
> >> releases. For more context, the description of PR
> >>  holds the details and
> waiting
> >> for the confirmation there prior to the merge.
> >>
> >> Regards,
> >> Apoorv Mittal
> >> +44 7721681581
> >>
> >>
> >> On Fri, Jan 5, 2024 at 10:22 PM Colin McCabe 
> wrote:
> >>
> >>> metadata is an internal gradle module. It is not used by clients. So I
> >>> don't see why you would want to publish it (unless I'm misunderstanding
> >>> something).
> >>>
> >>> best,
> >>> Colin
> >>>
> >>>
> >>> On Fri, Jan 5, 2024, at 10:05, Stanislav Kozlovski wrote:
> >>> > Thanks for reporting the blockers, folks. Good job finding.
> >>> >
> >>> > I have one ask - can anybody with Gradle expertise help review this
> small
> >>> > PR? https://github.com/apache/kafka/pull/15127 (+1, -1)
> >>> > In particular, we are wondering whether we need to publish module
> >>> metadata
> >>> > as part of the gradle publishing process.
> >>> >
> >>> >
> >>> > On Fri, Jan 5, 2024 at 3:56 PM Proven Provenzano
> >>> >  wrote:
> >>> >
> >>> >> We have potentially one more blocker
> >>> >> https://issues.apache.org/jira/browse/KAFKA-16082 which might
> cause a
> >>> data
> >>> >> loss scenario with JBOD in KRaft.
> >>> >> Initial analysis thought this is a problem and further review looks
> >>> like it
> >>> >> isn't but we are continuing to dig into the issue to ensure that it
> >>> isn't.
> >>> >> We would request feedback on the bug from anyone who is familiar
> with
> >>> this
> >>> >> code.
> >>> >>
> >>> >> --Proven
> >>> >>
> >>> >
> >>> >
> >>> > --
> >>> > Best,
> >>> > Stanislav
> >>>
>

Re: [DISCUSS] KIP-1014: Managing Unstable Metadata Versions in Apache Kafka

[Colin McCabe]

Hi Proven,

Thanks for the KIP. I think there is a need for this capability, for those of 
us who deploy from trunk (or branches dervied from trunk).

With regard to "unstable.metadata.versions.enable": is this going to be a 
documented configuration, or an internal one? I am guessing we want it to be 
documented, so that users can use it. If we do, we should probably also very 
prominently warn that THIS WILL BREAK UPGRADES FOR YOUR CLUSTER. That includes 
logging an ERROR message on startup, etc.

It would be good to document if a release can go out that contains "future MVs" 
that are unstable. Like can we make a 3.8 release that contains IBP_4_0_IV0 in 
MetadataVersion.java, as an unstable future MV? Personally I think the answer 
should be "yes," but with the usual caveats. When the actual 4.0 comes out, the 
unstable 4.0 MV that shipped in 3.8 probably won't work, and you won't be able 
to upgrade. (It was unstable, we told you not to use it.)

best,
Colin


On Fri, Jan 5, 2024, at 07:32, Proven Provenzano wrote:
> Hey folks,
>
> I am starting a discussion thread for managing unstable metadata 
> versions
> in Apache Kafka.
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1014%3A+Managing+Unstable+Metadata+Versions+in+Apache+Kafka
>
> This KIP is actually already implemented in 3.7 with PR
> https://github.com/apache/kafka/pull/14860.
> I have created this KIP to explain the motivation and how managing Metadata
> Versions is expected to work.
> Comments are greatly appreciated as this process can always be improved.
>
> -- 
> --Proven

Re: [DISCUSS] KIP-1015: Limit number of ssl connections in brokers

[Colin McCabe]

Hi zw,

As you yourself wrote in the rejected alternatives section, the existing 
listener-specific connection limit already lets administrators limit the number 
of SSL connections (assuming that one listener is SSL and another is not).

I don't understand the objection to just using that capability. You mention 
that "the protocol of the listener may change dynamically, and the limit of the 
listener also needs to be modified." But the connection limit is also dynamic, 
so that doesn't seem to be a problem. I didn't understand the objection that a 
per-listener limit doesn't allow you to "limit the ssl connections precisely" 
-- can you explain more?

Just as a general comment, I think the biggest use-case for mixed clusters that 
support both PLAINTEXT and SSL is when you have replication using PLAINTEXT and 
external connections using SSL. In that case, it's hardly worth having a 
separate limit for SSL, since the number of plaintext connections is bounded, 
and low. But perhaps your use-case is different. Do you really think you'll 
have both a lot of PLAINTEXT and a lot of SSL connections?

cheers,
Colin


On Sat, Jan 6, 2024, at 23:17, zw wrote:
> Hi all,
> I'd like to begin discussion on KIP-1015 which proposes to add new 
> configuration max.ssl.connections to limit number of ssl connections in 
> brokers.
> KIP link：
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1015%3A+Limit+number+of+ssl+connections+in+brokers
>
>
> Best,
> Jimmy Wang

Re: [DISCUSS] KIP-1012: The need for a Kafka 3.8.x release

[Colin McCabe]

On Fri, Jan 5, 2024, at 14:50, Greg Harris wrote:
> Hi Colin,
>
> Thanks for the quick reply!
>
>> If we cannot get KIP-853 done in time for 3.8, then we'd move to have 
>> another 3.x release.
>
> This is the crux of my concern, and this is satisfactory. This means
> that the "must-haves" are advisory only, and don't constitute a
> binding feature list or a feature-based release. Thank you!
>

Hi Greg,

I agree that you could say that the release is still time-based rather than 
feature-based since we will release on time even if we haven't finished 
everything. (And I think this is a good thing)

However, I wouldn't phrase the KIP as "advisory only" ... I would say it just 
spells out the conditions for the last 3.x release. Once those conditions are 
met, we will move on to 4.0.

>>The intention behind saying the timeline was "rough" was to make the obvious 
>>comment that if we shipped 3.7 in February rather than in January, it would 
>>still be in the spirit of following the KIP.
>> I kind of regret putting that comment in there now, since apparently there 
>> was a lot of misinterpretation! It didn't mean that the KIP itself was just 
>> a suggestion and not binding, or that we hadn't come to a consensus about 
>> 3.7 being the last release.
>
> I certainly did not get that sense from reading KIP-833 or the
> discussion thread after-the-fact. I was reading it using the
> contemporary interpretation you posted here: [1]. If I were voting on
> that KIP, I would not think that I was voting for 3.7 to be the last
> 3.x release.
>
>> To make this totally clear:
>> NO vote for this KIP ===> 3.7 is the last 3.x release, as per KIP-833.
>> YES vote for this KIP ===> 3.8 is the last 3.x release.
>
> This was not the precedent set by previous major version bump
> discussions. We need a mutual consensus to advance the major version,
> and lack of consensus means by default the next release will be a
> minor release.
> If KIP-833 actually changed this precedent, then I'll vote for this
> KIP just to restore the norm, as long as it is clear to everyone that
> after 3.8 the version will bump to 3.9 (and etc) unless another
> discussion reaches consensus about the project being ready for 4.0.
>

Just to be totally clear. KIP-1012 states that once we have a 3.x release with 
KIP-853 and a mechanism for configuring perpetual unclean leader election, we 
can move on to 4.0. There will not need to be additional discussions or KIPs to 
do that. If there is more stuff you feel is needed in 3.x to make migration 
possible, now is the time to speak up.

best,
Colin

>
> [1] https://lists.apache.org/thread/k3bbz7k0hsb4y0b2xn1lh7bjrmjowmcq
>
> Thanks!
> Greg
>
> On Fri, Jan 5, 2024 at 2:20 PM Colin McCabe  wrote:
>>
>> Hi Justine,
>>
>> If there are more things you think are needed in 3.8 for migration, let's 
>> discuss in the VOTE thread.
>>
>> best,
>> Colin
>>
>>
>> On Fri, Jan 5, 2024, at 09:23, Justine Olshan wrote:
>> > While I agree we should have this release and should vote on it soon, is it
>> > worth determining the exact items we need before we vote? Just so we are
>> > all in agreement?
>> > There is still some discussion on the road to 4.0 thread that may be worth
>> > having here.
>> >
>> > On Fri, Jan 5, 2024 at 1:25 AM Josep Prat 
>> > wrote:
>> >
>> >> Hi Colin,
>> >> Sorry for being quiet these last days (PTO).
>> >>
>> >> I will start the vote thread right away.
>> >>
>> >> Best,
>> >>
>> >>
>> >> ---
>> >> Josep Prat
>> >> Open Source Engineering Director, aivenjosep.p...@aiven.io   |
>> >> +491715557497 | aiven.io
>> >> Aiven Deutschland GmbH
>> >> Alexanderufer 3-7, 10117 Berlin
>> >> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>> >> Amtsgericht Charlottenburg, HRB 209739 B
>> >>
>> >> On Fri, Jan 5, 2024, 00:24 Colin McCabe  wrote:
>> >>
>> >> > Hi all,
>> >> >
>> >> > Since this has been open for a few weeks, are there any objections to
>> >> > starting the vote? What do you think, Josep?
>> >> >
>> >> > Since 3.8 is going to be the next release (according to the KIP) we
>> >> should
>> >> > really vote this in as soon as possible.
>> >> >
>> >> > Also, I created a wiki page about the 3.8 release with a tentative
>> >> > schedule.
>> >> > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.8.0
>> >> >
>> >> > Please let me know if these dates make sense -- they're just proposals
>> >> > right now.
>> >> >
>> >> > best,
>> >> > Colin
>> >> >
>> >> >
>> >> > On Thu, Dec 28, 2023, at 20:14, Colin McCabe wrote:
>> >> > > On Thu, Dec 28, 2023, at 18:17, Justine Olshan wrote:
>> >> > >> Hey Colin,
>> >> > >>
>> >> > >> Some folks were concerned about the lack of automatic unclean leader
>> >> > >> election. I mentioned that KIP-966 would actually be better with its
>> >> > >> aggressive recovery option.
>> >> > >> I think folks were hoping for some availability over durability
>> >> solution
>> >> > >> for KRaft, so if we don't do KIP-966 we should provide an alternative
>> >> > or be

Re: Apache Kafka 3.7.0 Release

[Colin McCabe]

Oops, hit send too soon. I see that #15127 was already merged. So we should no 
longer be publishing :metadata as part of the clients artifacts, right?

thanks,
Colin


On Mon, Jan 8, 2024, at 11:42, Colin McCabe wrote:
> Hi Apporv,
>
> Please remove the metadata module from any artifacts published for 
> clients. It is only used by the server.
>
> best,
> Colin
>
>
> On Sun, Jan 7, 2024, at 03:04, Apoorv Mittal wrote:
>> Hi Colin,
>> Thanks for the response. The only reason for asking the question of
>> publishing the metadata is because that's present in previous client
>> releases. For more context, the description of PR
>>  holds the details and waiting
>> for the confirmation there prior to the merge.
>>
>> Regards,
>> Apoorv Mittal
>> +44 7721681581
>>
>>
>> On Fri, Jan 5, 2024 at 10:22 PM Colin McCabe  wrote:
>>
>>> metadata is an internal gradle module. It is not used by clients. So I
>>> don't see why you would want to publish it (unless I'm misunderstanding
>>> something).
>>>
>>> best,
>>> Colin
>>>
>>>
>>> On Fri, Jan 5, 2024, at 10:05, Stanislav Kozlovski wrote:
>>> > Thanks for reporting the blockers, folks. Good job finding.
>>> >
>>> > I have one ask - can anybody with Gradle expertise help review this small
>>> > PR? https://github.com/apache/kafka/pull/15127 (+1, -1)
>>> > In particular, we are wondering whether we need to publish module
>>> metadata
>>> > as part of the gradle publishing process.
>>> >
>>> >
>>> > On Fri, Jan 5, 2024 at 3:56 PM Proven Provenzano
>>> >  wrote:
>>> >
>>> >> We have potentially one more blocker
>>> >> https://issues.apache.org/jira/browse/KAFKA-16082 which might cause a
>>> data
>>> >> loss scenario with JBOD in KRaft.
>>> >> Initial analysis thought this is a problem and further review looks
>>> like it
>>> >> isn't but we are continuing to dig into the issue to ensure that it
>>> isn't.
>>> >> We would request feedback on the bug from anyone who is familiar with
>>> this
>>> >> code.
>>> >>
>>> >> --Proven
>>> >>
>>> >
>>> >
>>> > --
>>> > Best,
>>> > Stanislav
>>>

Re: Apache Kafka 3.7.0 Release

[Colin McCabe]

Hi Apporv,

Please remove the metadata module from any artifacts published for clients. It 
is only used by the server.

best,
Colin


On Sun, Jan 7, 2024, at 03:04, Apoorv Mittal wrote:
> Hi Colin,
> Thanks for the response. The only reason for asking the question of
> publishing the metadata is because that's present in previous client
> releases. For more context, the description of PR
>  holds the details and waiting
> for the confirmation there prior to the merge.
>
> Regards,
> Apoorv Mittal
> +44 7721681581
>
>
> On Fri, Jan 5, 2024 at 10:22 PM Colin McCabe  wrote:
>
>> metadata is an internal gradle module. It is not used by clients. So I
>> don't see why you would want to publish it (unless I'm misunderstanding
>> something).
>>
>> best,
>> Colin
>>
>>
>> On Fri, Jan 5, 2024, at 10:05, Stanislav Kozlovski wrote:
>> > Thanks for reporting the blockers, folks. Good job finding.
>> >
>> > I have one ask - can anybody with Gradle expertise help review this small
>> > PR? https://github.com/apache/kafka/pull/15127 (+1, -1)
>> > In particular, we are wondering whether we need to publish module
>> metadata
>> > as part of the gradle publishing process.
>> >
>> >
>> > On Fri, Jan 5, 2024 at 3:56 PM Proven Provenzano
>> >  wrote:
>> >
>> >> We have potentially one more blocker
>> >> https://issues.apache.org/jira/browse/KAFKA-16082 which might cause a
>> data
>> >> loss scenario with JBOD in KRaft.
>> >> Initial analysis thought this is a problem and further review looks
>> like it
>> >> isn't but we are continuing to dig into the issue to ensure that it
>> isn't.
>> >> We would request feedback on the bug from anyone who is familiar with
>> this
>> >> code.
>> >>
>> >> --Proven
>> >>
>> >
>> >
>> > --
>> > Best,
>> > Stanislav
>>

Re: [VOTE] KIP-1012: The need for a Kafka 3.8.x release

[Colin McCabe]

On Mon, Jan 8, 2024, at 09:05, Chris Egerton wrote:
> Hi Josep,
>
> Thanks for the KIP! +1 (binding).
>
> One small nit: I don't think we necessarily have to hold ourselves to
> releasing 4.0.0 "3 to 4 months after 3.8 branch is created" (quoting the
> timeline section of the KIP). IMO it's fine to leave some wiggle room for
> the 4.0.0 release without codifying a timeline in this KIP. Maybe something
> like "some time after 3.8 branch is created" would be sufficient? Anyways,
> not a huge thing, I'm sure we'll all give 4.0.0 the flexibility it needs
> with the understanding that this KIP is more focused on 3.8.0 than 4.0.0.
>

Hmm... I don't see any obstacles in the path of releasing 4.0 after the 
traditional 4 months of development. Keep in mind, we're removing things from 
the code (the ability to support JDK8, ZooKeeper mode, etc.), not adding 
things. We already support JDK11 so saying that it's the minimum is a very 
quick change. Similarly, we already support KRaft so saying that it's the only 
mode should be a pretty quick change.

Also, we added a note that "the timeline is very rough" to KIP-833 and it caued 
all kinds of confusion. So overall I'd prefer to leave the language about 4.0 
unchanged.

best,
Colin

>
> Cheers,
>
> Chris
>
> On Mon, Jan 8, 2024 at 11:41 AM Greg Harris 
> wrote:
>
>> Thanks Josep for leading the KIP and building consensus on 3.8!
>>
>> +1 (binding)
>>
>> Greg
>>
>> On Sun, Jan 7, 2024 at 11:45 PM Josep Prat 
>> wrote:
>> >
>> > Hi all,
>> >
>> > Thanks for your comments,
>> > I reworded some parts of the KIP to express that:
>> > - The KIP is to agree that we need at least one more minor version in the
>> > 3.x series
>> > - Explicitly saying that the list of KIPs is not exhaustive and that if
>> > some are not done, we would need another minor version
>> > - Which are the KIPs/Features the community identified that should be
>> > present in a 3.x version so they can safely migrate to a potential 4.0
>> > version
>> > - The timeline of 4.0 (starting after branching, not after release)
>> > - Wording is now focusing more on the need to have at least another minor
>> > release in 3.x to enable and ease migration to a potential 4.0 version
>> >
>> > I always mention potential in terms of 4.0 as we don't know what will be
>> in
>> > there yet, and this KIP's scope is not meant to define this.
>> >
>> > Best,
>> >
>> > On Fri, Jan 5, 2024 at 10:46 PM Ismael Juma  wrote:
>> >
>> > > I agree with Colin. Also, 4.0 would be started after 3.8 is branched
>> (not
>> > > after 3.8.0 is released). The rest looks good.
>> > >
>> > > Thanks!
>> > >
>> > > Ismael
>> > >
>> > > On Fri, Jan 5, 2024 at 11:27 PM Colin McCabe 
>> wrote:
>> > >
>> > > > Hi,
>> > > >
>> > > > Thanks for calling the vote, Josep.
>> > > >
>> > > > I re-checked this, and saw something that we missed updating. One
>> thing
>> > > we
>> > > > talked about earlier is that KIP-966 is actually not required for
>> 3.8.
>> > > What
>> > > > is required is some way of enabling unclean leader election by
>> default in
>> > > > KRaft. (Which could be KIP-966, or something else). Please revise
>> this.
>> > > >
>> > > > best,
>> > > > Colin
>> > > >
>> > > > On Fri, Jan 5, 2024, at 02:50, Anton Agestam wrote:
>> > > > > +1 from me.
>> > > > >
>> > > > > Den fre 5 jan. 2024 kl 10:33 skrev Josep Prat
>> > > > :
>> > > > >
>> > > > >> Hi all,
>> > > > >>
>> > > > >> I'd like to start a vote on KIP-1012:
>> > > > >>
>> > > > >>
>> > > >
>> > >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1012%3A+The+need+for+a+Kafka+3.8.x+release
>> > > > >>
>> > > > >> Discussion thread is here:
>> > > > >> https://lists.apache.org/thread/kvdp2gmq5gd9txkvxh5vk3z2n55b04s5
>> > > > >>
>> > > > >> Thanks!
>> > > > >>
>> > > > >> ---
>> > > > >> Josep Prat
>> > > > >> Open Source Engineering Director, aivenjosep.p...@aiven.io   |
>> > > > >> +491715557497 | aiven.io
>> > > > >> Aiven Deutschland GmbH
>> > > > >> Alexanderufer 3-7, 10117 Berlin
>> > > > >> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>> > > > >> Amtsgericht Charlottenburg, HRB 209739 B
>> > > > >>
>> > > >
>> > >
>> >
>> >
>> > --
>> > [image: Aiven] 
>> >
>> > *Josep Prat*
>> > Open Source Engineering Director, *Aiven*
>> > josep.p...@aiven.io   |   +491715557497
>> > aiven.io    |   <
>> https://www.facebook.com/aivencloud>
>> >      <
>> https://twitter.com/aiven_io>
>> > *Aiven Deutschland GmbH*
>> > Alexanderufer 3-7, 10117 Berlin
>> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>> > Amtsgericht Charlottenburg, HRB 209739 B
>>

Re: [VOTE] KIP-1012: The need for a Kafka 3.8.x release

[Colin McCabe]

Thanks for the changes. +1 (binding)

best,
Colin

On Sun, Jan 7, 2024, at 23:43, Josep Prat wrote:
> Hi all,
>
> Thanks for your comments,
> I reworded some parts of the KIP to express that:
> - The KIP is to agree that we need at least one more minor version in the
> 3.x series
> - Explicitly saying that the list of KIPs is not exhaustive and that if
> some are not done, we would need another minor version
> - Which are the KIPs/Features the community identified that should be
> present in a 3.x version so they can safely migrate to a potential 4.0
> version
> - The timeline of 4.0 (starting after branching, not after release)
> - Wording is now focusing more on the need to have at least another minor
> release in 3.x to enable and ease migration to a potential 4.0 version
>
> I always mention potential in terms of 4.0 as we don't know what will be in
> there yet, and this KIP's scope is not meant to define this.
>
> Best,
>
> On Fri, Jan 5, 2024 at 10:46 PM Ismael Juma  wrote:
>
>> I agree with Colin. Also, 4.0 would be started after 3.8 is branched (not
>> after 3.8.0 is released). The rest looks good.
>>
>> Thanks!
>>
>> Ismael
>>
>> On Fri, Jan 5, 2024 at 11:27 PM Colin McCabe  wrote:
>>
>> > Hi,
>> >
>> > Thanks for calling the vote, Josep.
>> >
>> > I re-checked this, and saw something that we missed updating. One thing
>> we
>> > talked about earlier is that KIP-966 is actually not required for 3.8.
>> What
>> > is required is some way of enabling unclean leader election by default in
>> > KRaft. (Which could be KIP-966, or something else). Please revise this.
>> >
>> > best,
>> > Colin
>> >
>> > On Fri, Jan 5, 2024, at 02:50, Anton Agestam wrote:
>> > > +1 from me.
>> > >
>> > > Den fre 5 jan. 2024 kl 10:33 skrev Josep Prat
>> > :
>> > >
>> > >> Hi all,
>> > >>
>> > >> I'd like to start a vote on KIP-1012:
>> > >>
>> > >>
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1012%3A+The+need+for+a+Kafka+3.8.x+release
>> > >>
>> > >> Discussion thread is here:
>> > >> https://lists.apache.org/thread/kvdp2gmq5gd9txkvxh5vk3z2n55b04s5
>> > >>
>> > >> Thanks!
>> > >>
>> > >> ---
>> > >> Josep Prat
>> > >> Open Source Engineering Director, aivenjosep.p...@aiven.io   |
>> > >> +491715557497 | aiven.io
>> > >> Aiven Deutschland GmbH
>> > >> Alexanderufer 3-7, 10117 Berlin
>> > >> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>> > >> Amtsgericht Charlottenburg, HRB 209739 B
>> > >>
>> >
>>
>
>
> -- 
> [image: Aiven] 
>
> *Josep Prat*
> Open Source Engineering Director, *Aiven*
> josep.p...@aiven.io   |   +491715557497
> aiven.io    |   
>      
> *Aiven Deutschland GmbH*
> Alexanderufer 3-7, 10117 Berlin
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> Amtsgericht Charlottenburg, HRB 209739 B

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2548

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 461195 lines...]
Gradle Test Run :core:test > Gradle Test Executor 92 > ZooKeeperClientTest > 
testGetDataExistingZNode() STARTED

Gradle Test Run :core:test > Gradle Test Executor 92 > ZooKeeperClientTest > 
testGetDataExistingZNode() PASSED

Gradle Test Run :core:test > Gradle Test Executor 92 > ZooKeeperClientTest > 
testDeleteExistingZNode() STARTED

Gradle Test Run :core:test > Gradle Test Executor 92 > ZooKeeperClientTest > 
testDeleteExistingZNode() PASSED

Gradle Test Run :core:test > Gradle Test Executor 92 > ZooKeeperClientTest > 
testSessionExpiry() STARTED

Gradle Test Run :core:test > Gradle Test Executor 92 > ZooKeeperClientTest > 
testSessionExpiry() PASSED

Gradle Test Run :core:test > Gradle Test Executor 92 > ZooKeeperClientTest > 
testSetDataNonExistentZNode() STARTED

Gradle Test Run :core:test > Gradle Test Executor 92 > ZooKeeperClientTest > 
testSetDataNonExistentZNode() PASSED

Gradle Test Run :core:test > Gradle Test Executor 92 > ZooKeeperClientTest > 
testConnectionViaNettyClient() STARTED

Gradle Test Run :core:test > Gradle Test Executor 92 > ZooKeeperClientTest > 
testConnectionViaNettyClient() PASSED

Gradle Test Run :core:test > Gradle Test Executor 92 > ZooKeeperClientTest > 
testDeleteNonExistentZNode() STARTED

Gradle Test Run :core:test > Gradle Test Executor 92 > ZooKeeperClientTest > 
testDeleteNonExistentZNode() PASSED

Gradle Test Run :core:test > Gradle Test Executor 92 > ZooKeeperClientTest > 
testExistsExistingZNode() STARTED

Gradle Test Run :core:test > Gradle Test Executor 92 > ZooKeeperClientTest > 
testExistsExistingZNode() PASSED

Gradle Test Run :core:test > Gradle Test Executor 92 > ZooKeeperClientTest > 
testZooKeeperStateChangeRateMetrics() STARTED

Gradle Test Run :core:test > Gradle Test Executor 92 > ZooKeeperClientTest > 
testZooKeeperStateChangeRateMetrics() PASSED

Gradle Test Run :core:test > Gradle Test Executor 92 > ZooKeeperClientTest > 
testZNodeChangeHandlerForDeletion() STARTED

Gradle Test Run :core:test > Gradle Test Executor 92 > ZooKeeperClientTest > 
testZNodeChangeHandlerForDeletion() PASSED

Gradle Test Run :core:test > Gradle Test Executor 92 > ZooKeeperClientTest > 
testGetAclNonExistentZNode() STARTED

Gradle Test Run :core:test > Gradle Test Executor 92 > ZooKeeperClientTest > 
testGetAclNonExistentZNode() PASSED

Gradle Test Run :core:test > Gradle Test Executor 92 > ZooKeeperClientTest > 
testStateChangeHandlerForAuthFailure() STARTED

Gradle Test Run :core:test > Gradle Test Executor 92 > ZooKeeperClientTest > 
testStateChangeHandlerForAuthFailure() PASSED

Gradle Test Run :core:test > Gradle Test Executor 92 > 
AllocateProducerIdsRequestTest > testAllocateProducersIdSentToController() > 
testAllocateProducersIdSentToController [1] Type=Raft-Isolated, 
MetadataVersion=3.8-IV0, Security=PLAINTEXT STARTED

Gradle Test Run :core:test > Gradle Test Executor 92 > 
AllocateProducerIdsRequestTest > testAllocateProducersIdSentToController() > 
testAllocateProducersIdSentToController [1] Type=Raft-Isolated, 
MetadataVersion=3.8-IV0, Security=PLAINTEXT PASSED

Gradle Test Run :core:test > Gradle Test Executor 92 > 
AllocateProducerIdsRequestTest > testAllocateProducersIdSentToNonController() > 
testAllocateProducersIdSentToNonController [1] Type=Raft-Isolated, 
MetadataVersion=3.8-IV0, Security=PLAINTEXT STARTED

streams-4: SMOKE-TEST-CLIENT-CLOSED
streams-5: SMOKE-TEST-CLIENT-CLOSED
streams-0: SMOKE-TEST-CLIENT-CLOSED
streams-1: SMOKE-TEST-CLIENT-CLOSED
streams-3: SMOKE-TEST-CLIENT-CLOSED
streams-1: SMOKE-TEST-CLIENT-CLOSED
streams-2: SMOKE-TEST-CLIENT-CLOSED
streams-5: SMOKE-TEST-CLIENT-CLOSED
streams-2: SMOKE-TEST-CLIENT-CLOSED
streams-3: SMOKE-TEST-CLIENT-CLOSED
streams-3: SMOKE-TEST-CLIENT-CLOSED
streams-4: SMOKE-TEST-CLIENT-CLOSED
streams-0: SMOKE-TEST-CLIENT-CLOSED
streams-0: SMOKE-TEST-CLIENT-CLOSED
streams-2: SMOKE-TEST-CLIENT-CLOSED
streams-1: SMOKE-TEST-CLIENT-CLOSED
streams-6: SMOKE-TEST-CLIENT-CLOSED

> Task :streams:test

Gradle Test Run :streams:test > Gradle Test Executor 91 > 
RocksDBMetricsRecorderTest > 
shouldThrowIfDbToAddWasAlreadyAddedForOtherSegment() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 91 > 
RocksDBMetricsRecorderTest > 
shouldThrowIfDbToAddWasAlreadyAddedForOtherSegment() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 91 > 
RocksDBMetricsRecorderTest > 
shouldAddItselfToRecordingTriggerWhenFirstValueProvidersAreAddedAfterLastValueProvidersWereRemoved()
 STARTED

Gradle Test Run :streams:test > Gradle Test Executor 91 > 
RocksDBMetricsRecorderTest > 
shouldAddItselfToRecordingTriggerWhenFirstValueProvidersAreAddedAfterLastValueProvidersWereRemoved()
 PASSED

Gradle Test Run :streams:test > Gradle Test Executor 91 > 
RocksDBMetricsRecorderTest > shouldThrowIfValueProvidersToRemoveNotFound() 
STARTED

Gradle Test Run 

[jira] [Created] (KAFKA-16093) Spurious warnings logged to stderr about empty path annotations and providers not implementing provider interfaces

[Chris Egerton (Jira)]

Chris Egerton created KAFKA-16093:
-

 Summary: Spurious warnings logged to stderr about empty path 
annotations and providers not implementing provider interfaces
 Key: KAFKA-16093
 URL: https://issues.apache.org/jira/browse/KAFKA-16093
 Project: Kafka
  Issue Type: Improvement
  Components: connect
Affects Versions: 3.7.0, 3.8.0
Reporter: Chris Egerton
Assignee: Chris Egerton


Some warnings get logged to stderr on Connect startup. For example:
{quote}Jan 08, 2024 1:48:18 PM org.glassfish.jersey.internal.inject.Providers 
checkProviderRuntime

WARNING: A provider 
org.apache.kafka.connect.runtime.rest.resources.RootResource registered in 
SERVER runtime does not implement any provider interfaces applicable in the 
SERVER runtime. Due to constraint configuration problems the provider 
org.apache.kafka.connect.runtime.rest.resources.RootResource will be ignored. 

Jan 08, 2024 1:48:18 PM org.glassfish.jersey.internal.inject.Providers 
checkProviderRuntime

WARNING: A provider 
org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource registered 
in SERVER runtime does not implement any provider interfaces applicable in the 
SERVER runtime. Due to constraint configuration problems the provider 
org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource will be 
ignored. 

Jan 08, 2024 1:48:18 PM org.glassfish.jersey.internal.inject.Providers 
checkProviderRuntime

WARNING: A provider 
org.apache.kafka.connect.runtime.rest.resources.InternalConnectResource 
registered in SERVER runtime does not implement any provider interfaces 
applicable in the SERVER runtime. Due to constraint configuration problems the 
provider 
org.apache.kafka.connect.runtime.rest.resources.InternalConnectResource will be 
ignored. 

Jan 08, 2024 1:48:18 PM org.glassfish.jersey.internal.inject.Providers 
checkProviderRuntime

WARNING: A provider 
org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource 
registered in SERVER runtime does not implement any provider interfaces 
applicable in the SERVER runtime. Due to constraint configuration problems the 
provider 
org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource will 
be ignored. 

Jan 08, 2024 1:48:18 PM org.glassfish.jersey.internal.inject.Providers 
checkProviderRuntime

WARNING: A provider 
org.apache.kafka.connect.runtime.rest.resources.LoggingResource registered in 
SERVER runtime does not implement any provider interfaces applicable in the 
SERVER runtime. Due to constraint configuration problems the provider 
org.apache.kafka.connect.runtime.rest.resources.LoggingResource will be 
ignored. 

Jan 08, 2024 1:48:19 PM org.glassfish.jersey.internal.Errors logErrors

WARNING: The following warnings have been detected: WARNING: The (sub)resource 
method listLoggers in 
org.apache.kafka.connect.runtime.rest.resources.LoggingResource contains empty 
path annotation.

WARNING: The (sub)resource method listConnectors in 
org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource contains 
empty path annotation.

WARNING: The (sub)resource method createConnector in 
org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource contains 
empty path annotation.

WARNING: The (sub)resource method listConnectorPlugins in 
org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource 
contains empty path annotation.

WARNING: The (sub)resource method serverInfo in 
org.apache.kafka.connect.runtime.rest.resources.RootResource contains empty 
path annotation.
{quote}
These are benign, but can confuse and even frighten new users.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [PR] 3.7: Add documentation for Kafka 3.7 [kafka-site]

[via GitHub]

jolshan commented on PR #576:
URL: https://github.com/apache/kafka-site/pull/576#issuecomment-1881640198

   `+355,890`
   
     Is there a way to update just what is new 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: Apache Kafka 3.7.0 Release

[Stanislav Kozlovski]

I've almost got an RC candidate out to test. I just need a. review on the
kafka-site repo to update the website with the appropriate 3.7 subpages -
https://github.com/apache/kafka-site/pull/576/

On Mon, Jan 8, 2024 at 10:05 AM Lucas Brutschy
 wrote:

> Hi,
>
> we have fixed one memory leak in Kafka Streams, but there is still at
> least one missing in the code. I created
> https://issues.apache.org/jira/browse/KAFKA-16089 which is a blocker.
>
> Cheers,
> Lucas
>
> On Sun, Jan 7, 2024 at 12:05 PM Apoorv Mittal 
> wrote:
> >
> > Hi Colin,
> > Thanks for the response. The only reason for asking the question of
> > publishing the metadata is because that's present in previous client
> > releases. For more context, the description of PR
> >  holds the details and
> waiting
> > for the confirmation there prior to the merge.
> >
> > Regards,
> > Apoorv Mittal
> > +44 7721681581
> >
> >
> > On Fri, Jan 5, 2024 at 10:22 PM Colin McCabe  wrote:
> >
> > > metadata is an internal gradle module. It is not used by clients. So I
> > > don't see why you would want to publish it (unless I'm misunderstanding
> > > something).
> > >
> > > best,
> > > Colin
> > >
> > >
> > > On Fri, Jan 5, 2024, at 10:05, Stanislav Kozlovski wrote:
> > > > Thanks for reporting the blockers, folks. Good job finding.
> > > >
> > > > I have one ask - can anybody with Gradle expertise help review this
> small
> > > > PR? https://github.com/apache/kafka/pull/15127 (+1, -1)
> > > > In particular, we are wondering whether we need to publish module
> > > metadata
> > > > as part of the gradle publishing process.
> > > >
> > > >
> > > > On Fri, Jan 5, 2024 at 3:56 PM Proven Provenzano
> > > >  wrote:
> > > >
> > > >> We have potentially one more blocker
> > > >> https://issues.apache.org/jira/browse/KAFKA-16082 which might
> cause a
> > > data
> > > >> loss scenario with JBOD in KRaft.
> > > >> Initial analysis thought this is a problem and further review looks
> > > like it
> > > >> isn't but we are continuing to dig into the issue to ensure that it
> > > isn't.
> > > >> We would request feedback on the bug from anyone who is familiar
> with
> > > this
> > > >> code.
> > > >>
> > > >> --Proven
> > > >>
> > > >
> > > >
> > > > --
> > > > Best,
> > > > Stanislav
> > >
>


-- 
Best,
Stanislav

Re: [PR] 3.7: Add documentation for Kafka 3.7 [kafka-site]

[via GitHub]

stanislavkozlovski commented on PR #576:
URL: https://github.com/apache/kafka-site/pull/576#issuecomment-1881629932

   cc @showuon @C0urante @satishd could you perhaps help review?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] 3.7: Add documentation for Kafka 3.7 [kafka-site]

[via GitHub]

stanislavkozlovski commented on code in PR #576:
URL: https://github.com/apache/kafka-site/pull/576#discussion_r1445131803


##
37/javadoc/index.html:
##
@@ -0,0 +1,265 @@
+
+
+
+
+Overview (kafka 3.7.0-SNAPSHOT API)

Review Comment:
   ok figured out why. re-generated with the RC tag and updated the docs - 
   
![image](https://github.com/apache/kafka-site/assets/13639618/b64352f2-d8f9-48d4-9d20-bf9b1ba2db61)
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] 3.7: Add documentation for Kafka 3.7 [kafka-site]

[via GitHub]

stanislavkozlovski commented on code in PR #576:
URL: https://github.com/apache/kafka-site/pull/576#discussion_r1445116548


##
37/javadoc/index.html:
##
@@ -0,0 +1,265 @@
+
+
+
+
+Overview (kafka 3.7.0-SNAPSHOT API)

Review Comment:
   note that this is 3.7.0-SNAPSHOT
   similar to what happened in https://github.com/apache/kafka-site/pull/541 
and was later corrected in https://github.com/apache/kafka-site/pull/557
   
   I followed all the steps to the tee, so I am not sure where the disconnect 
is so that it creates it with the -SNAPSHOT tag.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[PR] 3.7: Add documentation for Kafka 3.7 [kafka-site]

[via GitHub]

stanislavkozlovski opened a new pull request, #576:
URL: https://github.com/apache/kafka-site/pull/576

   This patch adds the generated 3.7.0 release docs as mentioned in the 
[wiki](https://cwiki.apache.org/confluence/display/KAFKA/Release+Process#ReleaseProcess-Websiteupdateprocess)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[DISCUSS] KIP-853: KRaft Controller Membership Changes

[José Armando García Sancio]

Hi all,

KIP-853: KRaft Controller Membership Changes is ready for another
round of discussion.

There was a previous discussion thread at
https://lists.apache.org/thread/zb5l1fsqw9vj25zkmtnrk6xm7q3dkm1v

I have changed the KIP quite a bit since that discussion. The core
idea is still the same. I changed some of the details to be consistent
with some of the protocol changes to Kafka since the original KIP. I
also added a section that better describes the feature's UX.

KIP: https://cwiki.apache.org/confluence/x/nyH1D

Thanks. Your feedback is greatly appreciated!
-- 
-José

[jira] [Created] (KAFKA-16092) Queues for Kafka

[Andrew Schofield (Jira)]

Andrew Schofield created KAFKA-16092:


 Summary: Queues for Kafka
 Key: KAFKA-16092
 URL: https://issues.apache.org/jira/browse/KAFKA-16092
 Project: Kafka
  Issue Type: Improvement
Reporter: Andrew Schofield
Assignee: Andrew Schofield


This Jira tracks the development of KIP-932: 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.7 #52

[Apache Jenkins Server]

See 

[jira] [Resolved] (KAFKA-15846) Review consumer leave group request best effort and response handling

[Lianet Magrans (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lianet Magrans resolved KAFKA-15846.

Fix Version/s: (was: 3.8.0)
   Resolution: Duplicate

> Review consumer leave group request best effort and response handling
> -
>
> Key: KAFKA-15846
> URL: https://issues.apache.org/jira/browse/KAFKA-15846
> Project: Kafka
>  Issue Type: Sub-task
>  Components: clients, consumer
>Reporter: Lianet Magrans
>Priority: Major
>  Labels: kip-848, kip-848-client-support
>
> New consumer sends out a leave group request with a best effort approach. 
> Transitions to LEAVING to indicate the HB manager that the request must be 
> sent, but it does not do any response handling or retrying (note that the 
> response is still handled as any other HB response). After the first HB 
> manager poll iteration while on LEAVING, the consumer transitions into 
> UNSUBSCRIBE (no matter if the request was actually sent out or not, ex, due 
> to coordinator not known). Review if this is good enough as an effort to send 
> the request, and consider effect of responses that may be received and 
> processed when there are no longer relevant



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15542) Release member assignments on errors

[Lianet Magrans (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lianet Magrans resolved KAFKA-15542.

Fix Version/s: 3.7.0
   (was: 3.8.0)
   Resolution: Fixed

Fixed in https://github.com/apache/kafka/pull/14690

> Release member assignments on errors
> 
>
> Key: KAFKA-15542
> URL: https://issues.apache.org/jira/browse/KAFKA-15542
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients, consumer
>Reporter: Lianet Magrans
>Assignee: Lianet Magrans
>Priority: Major
>  Labels: kip-848, kip-848-client-support, kip-848-e2e, 
> kip-848-preview
> Fix For: 3.7.0
>
>
> Member should release assignment by triggering the onPartitionsLost flow from 
> the HB manager when errors occur (both fencing and unrecoverable errors)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-1012: The need for a Kafka 3.8.x release

[Chris Egerton]

Hi Josep,

Thanks for the KIP! +1 (binding).

One small nit: I don't think we necessarily have to hold ourselves to
releasing 4.0.0 "3 to 4 months after 3.8 branch is created" (quoting the
timeline section of the KIP). IMO it's fine to leave some wiggle room for
the 4.0.0 release without codifying a timeline in this KIP. Maybe something
like "some time after 3.8 branch is created" would be sufficient? Anyways,
not a huge thing, I'm sure we'll all give 4.0.0 the flexibility it needs
with the understanding that this KIP is more focused on 3.8.0 than 4.0.0.

Cheers,

Chris

On Mon, Jan 8, 2024 at 11:41 AM Greg Harris 
wrote:

> Thanks Josep for leading the KIP and building consensus on 3.8!
>
> +1 (binding)
>
> Greg
>
> On Sun, Jan 7, 2024 at 11:45 PM Josep Prat 
> wrote:
> >
> > Hi all,
> >
> > Thanks for your comments,
> > I reworded some parts of the KIP to express that:
> > - The KIP is to agree that we need at least one more minor version in the
> > 3.x series
> > - Explicitly saying that the list of KIPs is not exhaustive and that if
> > some are not done, we would need another minor version
> > - Which are the KIPs/Features the community identified that should be
> > present in a 3.x version so they can safely migrate to a potential 4.0
> > version
> > - The timeline of 4.0 (starting after branching, not after release)
> > - Wording is now focusing more on the need to have at least another minor
> > release in 3.x to enable and ease migration to a potential 4.0 version
> >
> > I always mention potential in terms of 4.0 as we don't know what will be
> in
> > there yet, and this KIP's scope is not meant to define this.
> >
> > Best,
> >
> > On Fri, Jan 5, 2024 at 10:46 PM Ismael Juma  wrote:
> >
> > > I agree with Colin. Also, 4.0 would be started after 3.8 is branched
> (not
> > > after 3.8.0 is released). The rest looks good.
> > >
> > > Thanks!
> > >
> > > Ismael
> > >
> > > On Fri, Jan 5, 2024 at 11:27 PM Colin McCabe 
> wrote:
> > >
> > > > Hi,
> > > >
> > > > Thanks for calling the vote, Josep.
> > > >
> > > > I re-checked this, and saw something that we missed updating. One
> thing
> > > we
> > > > talked about earlier is that KIP-966 is actually not required for
> 3.8.
> > > What
> > > > is required is some way of enabling unclean leader election by
> default in
> > > > KRaft. (Which could be KIP-966, or something else). Please revise
> this.
> > > >
> > > > best,
> > > > Colin
> > > >
> > > > On Fri, Jan 5, 2024, at 02:50, Anton Agestam wrote:
> > > > > +1 from me.
> > > > >
> > > > > Den fre 5 jan. 2024 kl 10:33 skrev Josep Prat
> > > > :
> > > > >
> > > > >> Hi all,
> > > > >>
> > > > >> I'd like to start a vote on KIP-1012:
> > > > >>
> > > > >>
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1012%3A+The+need+for+a+Kafka+3.8.x+release
> > > > >>
> > > > >> Discussion thread is here:
> > > > >> https://lists.apache.org/thread/kvdp2gmq5gd9txkvxh5vk3z2n55b04s5
> > > > >>
> > > > >> Thanks!
> > > > >>
> > > > >> ---
> > > > >> Josep Prat
> > > > >> Open Source Engineering Director, aivenjosep.p...@aiven.io   |
> > > > >> +491715557497 | aiven.io
> > > > >> Aiven Deutschland GmbH
> > > > >> Alexanderufer 3-7, 10117 Berlin
> > > > >> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > > >> Amtsgericht Charlottenburg, HRB 209739 B
> > > > >>
> > > >
> > >
> >
> >
> > --
> > [image: Aiven] 
> >
> > *Josep Prat*
> > Open Source Engineering Director, *Aiven*
> > josep.p...@aiven.io   |   +491715557497
> > aiven.io    |   <
> https://www.facebook.com/aivencloud>
> >      <
> https://twitter.com/aiven_io>
> > *Aiven Deutschland GmbH*
> > Alexanderufer 3-7, 10117 Berlin
> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > Amtsgericht Charlottenburg, HRB 209739 B
>

Re: [VOTE] KIP-971: Expose replication-offset-lag MirrorMaker2 metric

[Dániel Urbán]

Hi Elxan,
+1 (non-binding)
Thanks for the KIP, this will be a very useful metric for MM!
Daniel

Elxan Eminov  ezt írta (időpont: 2024. jan. 7., V,
2:17):

> Hi all,
> Bumping this for visibility
>
> On Wed, 3 Jan 2024 at 18:13, Elxan Eminov  wrote:
>
> > Hi All,
> > I'd like to initiate a vote for KIP-971.
> > This KIP is about adding a new metric to the MirrorSourceTask that tracks
> > the offset lag between a source and a target partition.
> >
> > KIP link:
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-971%3A+Expose+replication-offset-lag+MirrorMaker2+metric
> >
> > Discussion thread:
> > https://lists.apache.org/thread/gwq9jd75dnm8htmpqkn17bnks6h3wqwp
> >
> > Thanks!
> >
>

Re: [VOTE] KIP-1012: The need for a Kafka 3.8.x release

[Greg Harris]

Thanks Josep for leading the KIP and building consensus on 3.8!

+1 (binding)

Greg

On Sun, Jan 7, 2024 at 11:45 PM Josep Prat  wrote:
>
> Hi all,
>
> Thanks for your comments,
> I reworded some parts of the KIP to express that:
> - The KIP is to agree that we need at least one more minor version in the
> 3.x series
> - Explicitly saying that the list of KIPs is not exhaustive and that if
> some are not done, we would need another minor version
> - Which are the KIPs/Features the community identified that should be
> present in a 3.x version so they can safely migrate to a potential 4.0
> version
> - The timeline of 4.0 (starting after branching, not after release)
> - Wording is now focusing more on the need to have at least another minor
> release in 3.x to enable and ease migration to a potential 4.0 version
>
> I always mention potential in terms of 4.0 as we don't know what will be in
> there yet, and this KIP's scope is not meant to define this.
>
> Best,
>
> On Fri, Jan 5, 2024 at 10:46 PM Ismael Juma  wrote:
>
> > I agree with Colin. Also, 4.0 would be started after 3.8 is branched (not
> > after 3.8.0 is released). The rest looks good.
> >
> > Thanks!
> >
> > Ismael
> >
> > On Fri, Jan 5, 2024 at 11:27 PM Colin McCabe  wrote:
> >
> > > Hi,
> > >
> > > Thanks for calling the vote, Josep.
> > >
> > > I re-checked this, and saw something that we missed updating. One thing
> > we
> > > talked about earlier is that KIP-966 is actually not required for 3.8.
> > What
> > > is required is some way of enabling unclean leader election by default in
> > > KRaft. (Which could be KIP-966, or something else). Please revise this.
> > >
> > > best,
> > > Colin
> > >
> > > On Fri, Jan 5, 2024, at 02:50, Anton Agestam wrote:
> > > > +1 from me.
> > > >
> > > > Den fre 5 jan. 2024 kl 10:33 skrev Josep Prat
> > > :
> > > >
> > > >> Hi all,
> > > >>
> > > >> I'd like to start a vote on KIP-1012:
> > > >>
> > > >>
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1012%3A+The+need+for+a+Kafka+3.8.x+release
> > > >>
> > > >> Discussion thread is here:
> > > >> https://lists.apache.org/thread/kvdp2gmq5gd9txkvxh5vk3z2n55b04s5
> > > >>
> > > >> Thanks!
> > > >>
> > > >> ---
> > > >> Josep Prat
> > > >> Open Source Engineering Director, aivenjosep.p...@aiven.io   |
> > > >> +491715557497 | aiven.io
> > > >> Aiven Deutschland GmbH
> > > >> Alexanderufer 3-7, 10117 Berlin
> > > >> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > >> Amtsgericht Charlottenburg, HRB 209739 B
> > > >>
> > >
> >
>
>
> --
> [image: Aiven] 
>
> *Josep Prat*
> Open Source Engineering Director, *Aiven*
> josep.p...@aiven.io   |   +491715557497
> aiven.io    |   
>      
> *Aiven Deutschland GmbH*
> Alexanderufer 3-7, 10117 Berlin
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> Amtsgericht Charlottenburg, HRB 209739 B

[jira] [Resolved] (KAFKA-15719) KRaft support in OffsetsForLeaderEpochRequestTest

[Mickael Maison (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mickael Maison resolved KAFKA-15719.

Fix Version/s: 3.8.0
   Resolution: Fixed

> KRaft support in OffsetsForLeaderEpochRequestTest
> -
>
> Key: KAFKA-15719
> URL: https://issues.apache.org/jira/browse/KAFKA-15719
> Project: Kafka
>  Issue Type: Task
>  Components: core
>Reporter: Sameer Tejani
>Assignee: Zihao Lin
>Priority: Minor
>  Labels: kraft, kraft-test, newbie
> Fix For: 3.8.0
>
>
> The following tests in OffsetsForLeaderEpochRequestTest in 
> core/src/test/scala/unit/kafka/server/OffsetsForLeaderEpochRequestTest.scala 
> need to be updated to support KRaft
> 37 : def testOffsetsForLeaderEpochErrorCodes(): Unit = {
> 60 : def testCurrentEpochValidation(): Unit = {
> Scanned 127 lines. Found 0 KRaft tests out of 2 tests



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15515) Remove duplicated integration tests for new consumer

[Lianet Magrans (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lianet Magrans resolved KAFKA-15515.

Fix Version/s: (was: 4.0.0)
   Resolution: Not A Problem

PlaintextAsyncConsumer.scala was discarded before merging as it was not needed 
anymore (existing PlaintextConsumer.scala was parametrized)

> Remove duplicated integration tests for new consumer
> 
>
> Key: KAFKA-15515
> URL: https://issues.apache.org/jira/browse/KAFKA-15515
> Project: Kafka
>  Issue Type: Test
>  Components: clients, consumer, unit tests
>Reporter: Lianet Magrans
>Priority: Major
>  Labels: consumer-threading-refactor
>
> This task involves removing the temporary `PlaintextAsyncConsumer` file 
> containing duplicated integration tests for the new consumer. The copy was 
> generated to catch regressions and validate functionality in the new consumer 
> while in development. It should be deleted when the new consumer is fully 
> implemented and the existing integration tests (`PlaintextConsumerTest`) can 
> be executed for both implementations.
>  
> Context:
>  
> For the current KafkaConsumer, a set of integration tests exist in the file 
> PlaintextConsumerTest. Those tests cannot be executed as such for the new 
> consumer implementation for 2 main reasons
> - the new consumer is being developed as a new PrototypeAsyncConsumer class, 
> in parallel to the existing KafkaConsumer. 
> - the new consumer is under development, so it does not support all the 
> consumer functionality yet. 
>  
> In order to be able to run the subsets of tests that the new consumer 
> supports while the implementation completes, it was decided to :  
>  - to make a copy of the `PlaintextAsyncConsumer` class, named 
> PlaintextAsyncConsumer.
> - leave all the existing integration tests that cover the simple consumer 
> case unchanged, and disable the tests that are not yet supported by the new 
> consumer. Disabled tests will be enabled as the async consumer
> evolves.
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15725) KRaft support in FetchRequestTest

[Mickael Maison (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mickael Maison resolved KAFKA-15725.

Fix Version/s: 3.7.0
   Resolution: Fixed

> KRaft support in FetchRequestTest
> -
>
> Key: KAFKA-15725
> URL: https://issues.apache.org/jira/browse/KAFKA-15725
> Project: Kafka
>  Issue Type: Task
>  Components: core
>Reporter: Sameer Tejani
>Priority: Minor
>  Labels: kraft, kraft-test, newbie
> Fix For: 3.7.0
>
>
> The following tests in FetchRequestTest in 
> core/src/test/scala/unit/kafka/server/FetchRequestTest.scala need to be 
> updated to support KRaft
> 45 : def testBrokerRespectsPartitionsOrderAndSizeLimits(): Unit = {
> 147 : def testFetchRequestV4WithReadCommitted(): Unit = {
> 165 : def testFetchRequestToNonReplica(): Unit = {
> 195 : def testLastFetchedEpochValidation(): Unit = {
> 200 : def testLastFetchedEpochValidationV12(): Unit = {
> 247 : def testCurrentEpochValidation(): Unit = {
> 252 : def testCurrentEpochValidationV12(): Unit = {
> 295 : def testEpochValidationWithinFetchSession(): Unit = {
> 300 : def testEpochValidationWithinFetchSessionV12(): Unit = {
> 361 : def testDownConversionWithConnectionFailure(): Unit = {
> 428 : def testDownConversionFromBatchedToUnbatchedRespectsOffset(): Unit = {
> 509 : def testCreateIncrementalFetchWithPartitionsInErrorV12(): Unit = {
> 568 : def testFetchWithPartitionsWithIdError(): Unit = {
> 610 : def testZStdCompressedTopic(): Unit = {
> 657 : def testZStdCompressedRecords(): Unit = {
> Scanned 783 lines. Found 0 KRaft tests out of 15 tests



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16016) Migrate utility scripts to kafka codebase

[Vedarth Sharma (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vedarth Sharma resolved KAFKA-16016.

  Reviewer: Manikumar
Resolution: Fixed

> Migrate utility scripts to kafka codebase
> -
>
> Key: KAFKA-16016
> URL: https://issues.apache.org/jira/browse/KAFKA-16016
> Project: Kafka
>  Issue Type: Sub-task
>  Components: core
>Reporter: Vedarth Sharma
>Assignee: Vedarth Sharma
>Priority: Blocker
> Fix For: 3.8.0
>
>
> Migrate the logic implemented in golang to kafka codebase by creating a new 
> entrypoint for docker images



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[VOTE] KIP-995: Allow users to specify initial offsets while creating connectors

[Ashwin]

Hi All,

I would like to start  a vote on KIP-995.

https://cwiki.apache.org/confluence/display/KAFKA/KIP-995%3A+Allow+users+to+specify+initial+offsets+while+creating+connectors

Discussion thread -
https://lists.apache.org/thread/msorbr63scglf4484yq764v7klsj7c4j

Thanks!

Ashwin

[jira] [Resolved] (KAFKA-15325) Integrate topicId in OffsetFetch and OffsetCommit async consumer calls

[Lianet Magrans (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lianet Magrans resolved KAFKA-15325.

Resolution: Duplicate

> Integrate topicId in OffsetFetch and OffsetCommit async consumer calls
> --
>
> Key: KAFKA-15325
> URL: https://issues.apache.org/jira/browse/KAFKA-15325
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients, consumer
>Reporter: Lianet Magrans
>Assignee: Lianet Magrans
>Priority: Major
>  Labels: kip-848, kip-848-client-support
> Fix For: 3.8.0
>
>
> KIP-848 introduces support for topicIds in the OffsetFetch and OffsetCommit 
> APIs. The consumer calls to those APIs should be updated to include topicIds 
> when available.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Requesting permissions to contribute to Apache Kafka

[Abhinav Dixit]

Thanks Josep.

On Mon, Jan 8, 2024 at 7:38 PM Josep Prat 
wrote:

> Hi Abhinav,
>
> This one I could find :)
>
> You are now all set! Happy contributing!
>
> Best,
>
> On Mon, Jan 8, 2024 at 2:39 PM Abhinav Dixit 
> wrote:
>
> > Hi Josep,
> > Thanks for the quick response. My WIKI ID is adixit, sorry for the
> > confusion.
> > Thanks,
> > Abhinav
> >
> > On Mon, Jan 8, 2024 at 6:59 PM Josep Prat 
> > wrote:
> >
> > > Hi Abhinav,
> > > Thanks for your interest in Apache Kafka.
> > > I could set up your Jira account accordingly, but I can't find your
> Wiki
> > ID
> > > in the system. Can you confirm you created it and named it
> > > "adixitconfluent"?
> > >
> > > Best,
> > >
> > > On Mon, Jan 8, 2024 at 2:10 PM Abhinav Dixit
>  > >
> > > wrote:
> > >
> > > > Hi Team,
> > > > I am writing this email to request permissions to contribute to
> Apache
> > > > Kafka. Here are my details -
> > > > JIRA ID - adixitconfluent
> > > > WIKI ID - adixitconfluent
> > > >
> > > > Please let me know if anything is missing.
> > > > Thanks,
> > > > Abhinav
> > > >
> > >
> > >
> > > --
> > > [image: Aiven] 
> > >
> > > *Josep Prat*
> > > Open Source Engineering Director, *Aiven*
> > > josep.p...@aiven.io   |   +491715557497
> > > aiven.io    |   <
> > https://www.facebook.com/aivencloud
> > > >
> > >      <
> > > https://twitter.com/aiven_io>
> > > *Aiven Deutschland GmbH*
> > > Alexanderufer 3-7, 10117 Berlin
> > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > Amtsgericht Charlottenburg, HRB 209739 B
> > >
> >
>
>
> --
> [image: Aiven] 
>
> *Josep Prat*
> Open Source Engineering Director, *Aiven*
> josep.p...@aiven.io   |   +491715557497
> aiven.io    |    >
>      <
> https://twitter.com/aiven_io>
> *Aiven Deutschland GmbH*
> Alexanderufer 3-7, 10117 Berlin
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> Amtsgericht Charlottenburg, HRB 209739 B
>

Re: Requesting permissions to contribute to Apache Kafka

[Josep Prat]

Hi Abhinav,

This one I could find :)

You are now all set! Happy contributing!

Best,

On Mon, Jan 8, 2024 at 2:39 PM Abhinav Dixit 
wrote:

> Hi Josep,
> Thanks for the quick response. My WIKI ID is adixit, sorry for the
> confusion.
> Thanks,
> Abhinav
>
> On Mon, Jan 8, 2024 at 6:59 PM Josep Prat 
> wrote:
>
> > Hi Abhinav,
> > Thanks for your interest in Apache Kafka.
> > I could set up your Jira account accordingly, but I can't find your Wiki
> ID
> > in the system. Can you confirm you created it and named it
> > "adixitconfluent"?
> >
> > Best,
> >
> > On Mon, Jan 8, 2024 at 2:10 PM Abhinav Dixit  >
> > wrote:
> >
> > > Hi Team,
> > > I am writing this email to request permissions to contribute to Apache
> > > Kafka. Here are my details -
> > > JIRA ID - adixitconfluent
> > > WIKI ID - adixitconfluent
> > >
> > > Please let me know if anything is missing.
> > > Thanks,
> > > Abhinav
> > >
> >
> >
> > --
> > [image: Aiven] 
> >
> > *Josep Prat*
> > Open Source Engineering Director, *Aiven*
> > josep.p...@aiven.io   |   +491715557497
> > aiven.io    |   <
> https://www.facebook.com/aivencloud
> > >
> >      <
> > https://twitter.com/aiven_io>
> > *Aiven Deutschland GmbH*
> > Alexanderufer 3-7, 10117 Berlin
> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > Amtsgericht Charlottenburg, HRB 209739 B
> >
>


-- 
[image: Aiven] 

*Josep Prat*
Open Source Engineering Director, *Aiven*
josep.p...@aiven.io   |   +491715557497
aiven.io    |   
     
*Aiven Deutschland GmbH*
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
Amtsgericht Charlottenburg, HRB 209739 B

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2547

[Apache Jenkins Server]

See 

Re: Requesting permissions to contribute to Apache Kafka

[Abhinav Dixit]

Hi Josep,
Thanks for the quick response. My WIKI ID is adixit, sorry for the
confusion.
Thanks,
Abhinav

On Mon, Jan 8, 2024 at 6:59 PM Josep Prat 
wrote:

> Hi Abhinav,
> Thanks for your interest in Apache Kafka.
> I could set up your Jira account accordingly, but I can't find your Wiki ID
> in the system. Can you confirm you created it and named it
> "adixitconfluent"?
>
> Best,
>
> On Mon, Jan 8, 2024 at 2:10 PM Abhinav Dixit 
> wrote:
>
> > Hi Team,
> > I am writing this email to request permissions to contribute to Apache
> > Kafka. Here are my details -
> > JIRA ID - adixitconfluent
> > WIKI ID - adixitconfluent
> >
> > Please let me know if anything is missing.
> > Thanks,
> > Abhinav
> >
>
>
> --
> [image: Aiven] 
>
> *Josep Prat*
> Open Source Engineering Director, *Aiven*
> josep.p...@aiven.io   |   +491715557497
> aiven.io    |    >
>      <
> https://twitter.com/aiven_io>
> *Aiven Deutschland GmbH*
> Alexanderufer 3-7, 10117 Berlin
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> Amtsgericht Charlottenburg, HRB 209739 B
>

Re: Requesting permissions to contribute to Apache Kafka

[Josep Prat]

Hi Abhinav,
Thanks for your interest in Apache Kafka.
I could set up your Jira account accordingly, but I can't find your Wiki ID
in the system. Can you confirm you created it and named it
"adixitconfluent"?

Best,

On Mon, Jan 8, 2024 at 2:10 PM Abhinav Dixit 
wrote:

> Hi Team,
> I am writing this email to request permissions to contribute to Apache
> Kafka. Here are my details -
> JIRA ID - adixitconfluent
> WIKI ID - adixitconfluent
>
> Please let me know if anything is missing.
> Thanks,
> Abhinav
>


-- 
[image: Aiven] 

*Josep Prat*
Open Source Engineering Director, *Aiven*
josep.p...@aiven.io   |   +491715557497
aiven.io    |   
     
*Aiven Deutschland GmbH*
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
Amtsgericht Charlottenburg, HRB 209739 B

Jenkins build is unstable: Kafka » Kafka Branch Builder » 3.7 #51

[Apache Jenkins Server]

See 

Requesting permissions to contribute to Apache Kafka

[Abhinav Dixit]

Hi Team,
I am writing this email to request permissions to contribute to Apache
Kafka. Here are my details -
JIRA ID - adixitconfluent
WIKI ID - adixitconfluent

Please let me know if anything is missing.
Thanks,
Abhinav

[jira] [Created] (KAFKA-16091) Pipeline to run docker build test on a PR

[Vedarth Sharma (Jira)]

Vedarth Sharma created KAFKA-16091:
--

 Summary: Pipeline to run docker build test on a PR
 Key: KAFKA-16091
 URL: https://issues.apache.org/jira/browse/KAFKA-16091
 Project: Kafka
  Issue Type: Sub-task
Reporter: Vedarth Sharma
Assignee: Vedarth Sharma


Create a new pipeline that will allow the capability to run the docker sanity 
tests on a PR



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16090) Refactor call to storage tool from kafka docker wrapper

[Vedarth Sharma (Jira)]

Vedarth Sharma created KAFKA-16090:
--

 Summary: Refactor call to storage tool from kafka docker wrapper
 Key: KAFKA-16090
 URL: https://issues.apache.org/jira/browse/KAFKA-16090
 Project: Kafka
  Issue Type: Sub-task
Reporter: Vedarth Sharma
Assignee: Vedarth Sharma


Once rewrite of kafka storage tool is done, refactor how we are calling storage 
tool from kafka docker wrapper



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Apache Kafka 3.7.0 Release

[Lucas Brutschy]

Hi,

we have fixed one memory leak in Kafka Streams, but there is still at
least one missing in the code. I created
https://issues.apache.org/jira/browse/KAFKA-16089 which is a blocker.

Cheers,
Lucas

On Sun, Jan 7, 2024 at 12:05 PM Apoorv Mittal  wrote:
>
> Hi Colin,
> Thanks for the response. The only reason for asking the question of
> publishing the metadata is because that's present in previous client
> releases. For more context, the description of PR
>  holds the details and waiting
> for the confirmation there prior to the merge.
>
> Regards,
> Apoorv Mittal
> +44 7721681581
>
>
> On Fri, Jan 5, 2024 at 10:22 PM Colin McCabe  wrote:
>
> > metadata is an internal gradle module. It is not used by clients. So I
> > don't see why you would want to publish it (unless I'm misunderstanding
> > something).
> >
> > best,
> > Colin
> >
> >
> > On Fri, Jan 5, 2024, at 10:05, Stanislav Kozlovski wrote:
> > > Thanks for reporting the blockers, folks. Good job finding.
> > >
> > > I have one ask - can anybody with Gradle expertise help review this small
> > > PR? https://github.com/apache/kafka/pull/15127 (+1, -1)
> > > In particular, we are wondering whether we need to publish module
> > metadata
> > > as part of the gradle publishing process.
> > >
> > >
> > > On Fri, Jan 5, 2024 at 3:56 PM Proven Provenzano
> > >  wrote:
> > >
> > >> We have potentially one more blocker
> > >> https://issues.apache.org/jira/browse/KAFKA-16082 which might cause a
> > data
> > >> loss scenario with JBOD in KRaft.
> > >> Initial analysis thought this is a problem and further review looks
> > like it
> > >> isn't but we are continuing to dig into the issue to ensure that it
> > isn't.
> > >> We would request feedback on the bug from anyone who is familiar with
> > this
> > >> code.
> > >>
> > >> --Proven
> > >>
> > >
> > >
> > > --
> > > Best,
> > > Stanislav
> >

[jira] [Created] (KAFKA-16089) Kafka Streams still leaking memory in 3.7

[Lucas Brutschy (Jira)]

Lucas Brutschy created KAFKA-16089:
--

 Summary: Kafka Streams still leaking memory in 3.7
 Key: KAFKA-16089
 URL: https://issues.apache.org/jira/browse/KAFKA-16089
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 3.7.0
Reporter: Lucas Brutschy
 Fix For: 3.7.0
 Attachments: graphviz (1).svg

In 
[https://github.com/apache/kafka/commit/58d6d2e5922df60cdc5c5c1bcd1c2d82dd2b71e2]
 a leak was fixed in the release candidate for 3.7.

 

However, Kafka Streams still seems to be leaking memory (just slower) after the 
fix.

 

Attached is the `jeprof` output right before a crash after ~11 hours.

 

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)