Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2911

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 475363 lines...]
[2024-05-17T02:50:57.871Z] Gradle Test Run :core:test > Gradle Test Executor 99 
> KafkaZkClientTest > testTopicAssignmentMethods() STARTED
[2024-05-17T02:50:58.948Z] 
[2024-05-17T02:50:58.948Z] Gradle Test Run :core:test > Gradle Test Executor 99 
> KafkaZkClientTest > testTopicAssignmentMethods() PASSED
[2024-05-17T02:50:58.948Z] 
[2024-05-17T02:50:58.948Z] Gradle Test Run :core:test > Gradle Test Executor 99 
> KafkaZkClientTest > testConnectionViaNettyClient() STARTED
[2024-05-17T02:50:58.948Z] 
[2024-05-17T02:50:58.948Z] Gradle Test Run :core:test > Gradle Test Executor 99 
> KafkaZkClientTest > testConnectionViaNettyClient() PASSED
[2024-05-17T02:50:58.948Z] 
[2024-05-17T02:50:58.948Z] Gradle Test Run :core:test > Gradle Test Executor 99 
> KafkaZkClientTest > testPropagateIsrChanges() STARTED
[2024-05-17T02:51:00.026Z] 
[2024-05-17T02:51:00.026Z] Gradle Test Run :core:test > Gradle Test Executor 99 
> KafkaZkClientTest > testPropagateIsrChanges() PASSED
[2024-05-17T02:51:00.026Z] 
[2024-05-17T02:51:00.026Z] Gradle Test Run :core:test > Gradle Test Executor 99 
> KafkaZkClientTest > testNoEntityConfigUpdateWithNoController() STARTED
[2024-05-17T02:51:00.026Z] 
[2024-05-17T02:51:00.026Z] Gradle Test Run :core:test > Gradle Test Executor 99 
> KafkaZkClientTest > testNoEntityConfigUpdateWithNoController() PASSED
[2024-05-17T02:51:00.026Z] 
[2024-05-17T02:51:00.026Z] Gradle Test Run :core:test > Gradle Test Executor 99 
> KafkaZkClientTest > testControllerEpochMethods() STARTED
[2024-05-17T02:51:00.026Z] 
[2024-05-17T02:51:00.026Z] Gradle Test Run :core:test > Gradle Test Executor 99 
> KafkaZkClientTest > testControllerEpochMethods() PASSED
[2024-05-17T02:51:00.026Z] 
[2024-05-17T02:51:00.026Z] Gradle Test Run :core:test > Gradle Test Executor 99 
> KafkaZkClientTest > testDeleteRecursive() STARTED
[2024-05-17T02:51:01.279Z] 
[2024-05-17T02:51:01.279Z] Gradle Test Run :core:test > Gradle Test Executor 99 
> KafkaZkClientTest > testDeleteRecursive() PASSED
[2024-05-17T02:51:01.279Z] 
[2024-05-17T02:51:01.279Z] Gradle Test Run :core:test > Gradle Test Executor 99 
> KafkaZkClientTest > testGetTopicPartitionStates() STARTED
[2024-05-17T02:51:01.279Z] 
[2024-05-17T02:51:01.279Z] Gradle Test Run :core:test > Gradle Test Executor 99 
> KafkaZkClientTest > testGetTopicPartitionStates() PASSED
[2024-05-17T02:51:01.279Z] 
[2024-05-17T02:51:01.279Z] Gradle Test Run :core:test > Gradle Test Executor 99 
> KafkaZkClientTest > testCreateConfigChangeNotification() STARTED
[2024-05-17T02:51:01.279Z] 
[2024-05-17T02:51:01.279Z] Gradle Test Run :core:test > Gradle Test Executor 99 
> KafkaZkClientTest > testCreateConfigChangeNotification() PASSED
[2024-05-17T02:51:01.279Z] 
[2024-05-17T02:51:01.279Z] Gradle Test Run :core:test > Gradle Test Executor 99 
> KafkaZkClientTest > testDelegationTokenMethods() STARTED
[2024-05-17T02:51:01.279Z] 
[2024-05-17T02:51:01.279Z] Gradle Test Run :core:test > Gradle Test Executor 99 
> KafkaZkClientTest > testDelegationTokenMethods() PASSED
[2024-05-17T02:51:01.279Z] 
[2024-05-17T02:51:01.279Z] Gradle Test Run :core:test > Gradle Test Executor 99 
> ReassignPartitionsZNodeTest > testDecodeInvalidJson() STARTED
[2024-05-17T02:51:01.279Z] 
[2024-05-17T02:51:01.279Z] Gradle Test Run :core:test > Gradle Test Executor 99 
> ReassignPartitionsZNodeTest > testDecodeInvalidJson() PASSED
[2024-05-17T02:51:01.279Z] 
[2024-05-17T02:51:01.279Z] Gradle Test Run :core:test > Gradle Test Executor 99 
> ReassignPartitionsZNodeTest > testEncode() STARTED
[2024-05-17T02:51:01.279Z] 
[2024-05-17T02:51:01.279Z] Gradle Test Run :core:test > Gradle Test Executor 99 
> ReassignPartitionsZNodeTest > testEncode() PASSED
[2024-05-17T02:51:01.279Z] 
[2024-05-17T02:51:01.279Z] Gradle Test Run :core:test > Gradle Test Executor 99 
> ReassignPartitionsZNodeTest > testDecodeValidJson() STARTED
[2024-05-17T02:51:01.279Z] 
[2024-05-17T02:51:01.279Z] Gradle Test Run :core:test > Gradle Test Executor 99 
> ReassignPartitionsZNodeTest > testDecodeValidJson() PASSED
[2024-05-17T02:51:01.279Z] 
[2024-05-17T02:51:01.279Z] Gradle Test Run :core:test > Gradle Test Executor 99 
> ZkMigrationIntegrationTest > testMigrateTopicDeletions(ClusterInstance) > 
testMigrateTopicDeletions [1] Type=ZK, 
MetadataVersion=3.4-IV0,Security=PLAINTEXT STARTED
[2024-05-17T02:51:30.151Z] 
[2024-05-17T02:51:30.151Z] Gradle Test Run :core:test > Gradle Test Executor 99 
> ZkMigrationIntegrationTest > testMigrateTopicDeletions(ClusterInstance) > 
testMigrateTopicDeletions [1] Type=ZK, 
MetadataVersion=3.4-IV0,Security=PLAINTEXT SKIPPED
[2024-05-17T02:51:30.151Z] 
[2024-05-17T02:51:30.151Z] Gradle Test Run :core:test > Gradle Test Executor 99 
> ZkMigrationIntegrationTest > testMigrateTopicDeletions(ClusterInstance) > 
testMigrateTopicDeletions [2] Type=ZK, 

[jira] [Created] (KAFKA-16787) Remove TRACE level logging from AsyncKafkaConsumer hot path

[Kirk True (Jira)]

Kirk True created KAFKA-16787:
-

 Summary: Remove TRACE level logging from AsyncKafkaConsumer hot 
path
 Key: KAFKA-16787
 URL: https://issues.apache.org/jira/browse/KAFKA-16787
 Project: Kafka
  Issue Type: Improvement
  Components: clients, consumer, logging
Affects Versions: 3.7.0
Reporter: Kirk True
 Fix For: 3.8.0






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: please let me subscribe

[Greg Harris]

Hi Harry,

Thanks for your interest in Apache Kafka. You can see how to subscribe
to the mailing list(s) here: https://kafka.apache.org/contact
Please note that the email for subscribing/unsubscribing is different
from the one used for sending and receiving emails once you're on the
list.
Also, subscribing needs a follow-up email to confirm the subscription,
see here: https://www.apache.org/foundation/mailinglists#subscribing

Hope this helps!
Greg

On Thu, May 16, 2024 at 5:51 AM Harry Fallows
 wrote:
>
> please
>
> Harry Fallows
>
> Software Engineer
>
> Email: hfall...@thoughtmachine.net
>
> Web: www.thoughtmachine.net
>
>
> 7 Herbrand Street | London | WC1N 1EX
>
>
>
> Data Classification: Internal
>
> --
> Thought Machine Group Limited, a company registered in England & Wales.
> Registered number: 4277.
> Registered Office: 5 New Street Square,
> London EC4A 3TW
> .
>
>
> The content of this email is confidential and intended for the recipient
> specified in message only. It is strictly forbidden to share any part of
> this message with any third party, without a written consent of the sender.
> If you received this message by mistake, please reply to this message and
> follow with its deletion, so that we can ensure such a mistake does not
> occur in the future.

Re: [DISCUSS] KIP-1046: Expose producer.id.expiration.check.interval.ms as dynamic broker configuration

[Jorge Esteban Quilcate Otoya]

Thanks Justine. I have updated the KIP with the configuration details.

On Thu, 16 May 2024 at 21:14, Justine Olshan 
wrote:

> Hey Jorge,
>
> Thanks for the KIP. I didn't realize until I read the details that this
> configuration is currently not public at all. I think it is still ok that
> we are exposing the value though. Can we just include some information
> about the current default, the documentation etc that is already defined as
> this will now become part of the public documentation?
>
> Thanks,
> Justine
>
> On Thu, May 16, 2024 at 10:23 AM Jorge Esteban Quilcate Otoya <
> quilcate.jo...@gmail.com> wrote:
>
> > Hi dev team,
> >
> > I'd like to start a discussion thread for KIP-1046:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1046%3A+Expose+producer.id.expiration.check.interval.ms+as+dynamic+broker+configuration
> >
> > This KIP aims to align how tuning configurations for Producer ID
> expiration
> > checks are exposed.
> >
> > Looking forward to your feedback.
> >
> > Cheers,
> > Jorge.
> >
>

Re: [VOTE] KIP-989: RocksDB Iterator Metrics

[Nick Telford]

Oh shoot, you're right. I miscounted.

The vote remains open.

On Thu, 16 May 2024, 20:11 Josep Prat,  wrote:

> Hi Nick,
> I think you need one more day to reach the 72 hours. You opened the vote on
> the 14th, right?
>
> Best,
>
> 
>
> Josep Prat
> Open Source Engineering Director, aivenjosep.p...@aiven.io   |
> +491715557497 | aiven.io
> Aiven Deutschland GmbH
> Alexanderufer 3-7, 10117 Berlin
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> Amtsgericht Charlottenburg, HRB 209739 B
>
> On Thu, May 16, 2024, 19:40 Nick Telford  wrote:
>
> > Hi everyone,
> >
> > With 3 binding votes and no objections, the vote passes.
> >
> > KIP-989 is adopted.
> >
> > Cheers,
> > Nick
> >
> > On Wed, 15 May 2024 at 03:41, Sophie Blee-Goldman  >
> > wrote:
> >
> > > +1 (binding)
> > >
> > > Thanks!
> > >
> > > On Tue, May 14, 2024 at 6:58 PM Matthias J. Sax 
> > wrote:
> > >
> > > > +1 (binding)
> > > >
> > > > On 5/14/24 9:19 AM, Lucas Brutschy wrote:
> > > > > Hi Nick!
> > > > >
> > > > > Thanks for the KIP.
> > > > >
> > > > > +1 (binding)
> > > > >
> > > > > On Tue, May 14, 2024 at 5:16 PM Nick Telford <
> nick.telf...@gmail.com
> > >
> > > > wrote:
> > > > >>
> > > > >> Hi everyone,
> > > > >>
> > > > >> I'd like to call a vote on the Kafka Streams KIP-989: RocksDB
> > Iterator
> > > > >> Metrics:
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-989%3A+RocksDB+Iterator+Metrics
> > > > >>
> > > > >> All of the points in the discussion thread have now been
> addressed.
> > > > >>
> > > > >> Regards,
> > > > >>
> > > > >> Nick
> > > >
> > >
> >
>

Re: [VOTE] KIP-989: RocksDB Iterator Metrics

[Josep Prat]

Hi Nick,
I think you need one more day to reach the 72 hours. You opened the vote on
the 14th, right?

Best,



Josep Prat
Open Source Engineering Director, aivenjosep.p...@aiven.io   |
+491715557497 | aiven.io
Aiven Deutschland GmbH
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
Amtsgericht Charlottenburg, HRB 209739 B

On Thu, May 16, 2024, 19:40 Nick Telford  wrote:

> Hi everyone,
>
> With 3 binding votes and no objections, the vote passes.
>
> KIP-989 is adopted.
>
> Cheers,
> Nick
>
> On Wed, 15 May 2024 at 03:41, Sophie Blee-Goldman 
> wrote:
>
> > +1 (binding)
> >
> > Thanks!
> >
> > On Tue, May 14, 2024 at 6:58 PM Matthias J. Sax 
> wrote:
> >
> > > +1 (binding)
> > >
> > > On 5/14/24 9:19 AM, Lucas Brutschy wrote:
> > > > Hi Nick!
> > > >
> > > > Thanks for the KIP.
> > > >
> > > > +1 (binding)
> > > >
> > > > On Tue, May 14, 2024 at 5:16 PM Nick Telford  >
> > > wrote:
> > > >>
> > > >> Hi everyone,
> > > >>
> > > >> I'd like to call a vote on the Kafka Streams KIP-989: RocksDB
> Iterator
> > > >> Metrics:
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-989%3A+RocksDB+Iterator+Metrics
> > > >>
> > > >> All of the points in the discussion thread have now been addressed.
> > > >>
> > > >> Regards,
> > > >>
> > > >> Nick
> > >
> >
>

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.7 #156

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-989: RocksDB Iterator Metrics

[Matthias J. Sax]

SGTM to use nano-seconds across the board.

On 5/16/24 7:12 AM, Nick Telford wrote:

Actually, one other point: our existing state store operation metrics are
measured in nanoseconds[1].

Should iterator-duration-(avg|max) also be measured in nanoseconds, for
consistency, or should we keep them milliseconds, as the KIP currently
states?

1:
https://docs.confluent.io/platform/current/streams/monitoring.html#state-store-metrics

On Thu, 16 May 2024 at 12:15, Nick Telford  wrote:


Good point! I've updated it to "Improved StateStore Iterator metrics for
detecting leaks" - let me know if you have a better suggestion.

This should affect the voting imo, as nothing of substance has changed.

Regards,
Nick

On Thu, 16 May 2024 at 01:39, Sophie Blee-Goldman 
wrote:


One quick thing -- can you update the title of this KIP to reflect the
decision to implement these metrics for all state stores implementations
rather than just RocksDB?


On Tue, May 14, 2024 at 1:36 PM Nick Telford 
wrote:


Woops! Thanks for the catch Lucas. Given this was just a typo, I don't
think this affects the voting.

Cheers,
Nick

On Tue, 14 May 2024 at 18:06, Lucas Brutschy 
wrote:


Hi Nick,

you are still referring to oldest-open-iterator-age-ms in the
`Proposed Changes` section.

Cheers,
Lucas

On Thu, May 2, 2024 at 4:00 PM Lucas Brutschy 


wrote:


Hi Nick!

I agree, the age variant is a bit nicer since the semantics are very
clear from the name. If you'd rather go for the simple

implementation,

how about calling it `oldest-iterator-open-since-ms`? I believe this
could be understood without docs. Either way, I think we should be
able to open the vote for this KIP because nobody raised any major /
blocking concerns.

Looking forward to getting this voted on soon!

Cheers
Lucas

On Sun, Mar 31, 2024 at 5:23 PM Nick Telford <

nick.telf...@gmail.com>

wrote:


Hi Matthias,


For the oldest iterator metric, I would propose something simple

like

`iterator-opened-ms` and it would just be the actual timestamp

when

the

iterator was opened. I don't think we need to compute the actual

age,

but user can to this computation themselves?


That works for me; it's easier to implement like that :-D I'm a

little

concerned that the name "iterator-opened-ms" may not be obvious

enough

without reading the docs.


If we think reporting the age instead of just the timestamp is

better, I

would propose `iterator-max-age-ms`. I should be sufficient to

call

out

(as it's kinda "obvious" anyway) that the metric applies to open
iterator only.


While I think it's preferable to record the timestamp, rather than

the

age,

this does have the benefit of a more obvious metric name.


Nit: the KIP says it's a store-level metric, but I think it

would

be

good to say explicitly that it's recorded with DEBUG level only?


Yes, I've already updated the KIP with this information in the

table.


Regards,

Nick

On Sun, 31 Mar 2024 at 10:53, Matthias J. Sax 

wrote:



The time window thing was just an idea. Happy to drop it.

For the oldest iterator metric, I would propose something simple

like

`iterator-opened-ms` and it would just be the actual timestamp

when

the

iterator was opened. I don't think we need to compute the actual

age,

but user can to this computation themselves?

If we think reporting the age instead of just the timestamp is

better, I

would propose `iterator-max-age-ms`. I should be sufficient to

call

out

(as it's kinda "obvious" anyway) that the metric applies to open
iterator only.

And yes, I was hoping that the code inside MetereXxxStore might

already

be setup in a way that custom stores would inherit the iterator

metrics

automatically -- I am just not sure, and left it as an exercise

for

somebody to confirm :)


Nit: the KIP says it's a store-level metric, but I think it

would

be

good to say explicitly that it's recorded with DEBUG level only?



-Matthias


On 3/28/24 2:52 PM, Nick Telford wrote:

Quick addendum:

My suggested metric "oldest-open-iterator-age-seconds" should

be

"oldest-open-iterator-age-ms". Milliseconds is obviously a

better

granularity for such a metric.

Still accepting suggestions for a better name.

On Thu, 28 Mar 2024 at 13:41, Nick Telford <

nick.telf...@gmail.com



wrote:



Hi everyone,

Sorry for leaving this for so long. So much for "3 weeks

until

KIP

freeze"!


On Sophie's comments:
1. Would Matthias's suggestion of a separate metric tracking

the

age of

the oldest open iterator (within the tag set) satisfy this?

That

way we

can

keep iterator-duration-(avg|max) for closed iterators, which

can

be

useful

for performance debugging for iterators that don't leak. I'm

not

sure

what

we'd call this metric, maybe:

"oldest-open-iterator-age-seconds"?

Seems

like a mouthful.

2. You're right, it makes more sense to provide
iterator-duration-(avg|max). Honestly, I can't remember why I

had

"total"

before, or why I was computing a rate-of-change over it.

3, 4, 5, 

Re: [DISCUSS] KIP-1046: Expose producer.id.expiration.check.interval.ms as dynamic broker configuration

[Justine Olshan]

Hey Jorge,

Thanks for the KIP. I didn't realize until I read the details that this
configuration is currently not public at all. I think it is still ok that
we are exposing the value though. Can we just include some information
about the current default, the documentation etc that is already defined as
this will now become part of the public documentation?

Thanks,
Justine

On Thu, May 16, 2024 at 10:23 AM Jorge Esteban Quilcate Otoya <
quilcate.jo...@gmail.com> wrote:

> Hi dev team,
>
> I'd like to start a discussion thread for KIP-1046:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1046%3A+Expose+producer.id.expiration.check.interval.ms+as+dynamic+broker+configuration
>
> This KIP aims to align how tuning configurations for Producer ID expiration
> checks are exposed.
>
> Looking forward to your feedback.
>
> Cheers,
> Jorge.
>

Re: [VOTE] KIP-989: RocksDB Iterator Metrics

[Nick Telford]

Hi everyone,

With 3 binding votes and no objections, the vote passes.

KIP-989 is adopted.

Cheers,
Nick

On Wed, 15 May 2024 at 03:41, Sophie Blee-Goldman 
wrote:

> +1 (binding)
>
> Thanks!
>
> On Tue, May 14, 2024 at 6:58 PM Matthias J. Sax  wrote:
>
> > +1 (binding)
> >
> > On 5/14/24 9:19 AM, Lucas Brutschy wrote:
> > > Hi Nick!
> > >
> > > Thanks for the KIP.
> > >
> > > +1 (binding)
> > >
> > > On Tue, May 14, 2024 at 5:16 PM Nick Telford 
> > wrote:
> > >>
> > >> Hi everyone,
> > >>
> > >> I'd like to call a vote on the Kafka Streams KIP-989: RocksDB Iterator
> > >> Metrics:
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-989%3A+RocksDB+Iterator+Metrics
> > >>
> > >> All of the points in the discussion thread have now been addressed.
> > >>
> > >> Regards,
> > >>
> > >> Nick
> >
>

[DISCUSS] KIP-1046: Expose producer.id.expiration.check.interval.ms as dynamic broker configuration

[Jorge Esteban Quilcate Otoya]

Hi dev team,

I'd like to start a discussion thread for KIP-1046:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1046%3A+Expose+producer.id.expiration.check.interval.ms+as+dynamic+broker+configuration

This KIP aims to align how tuning configurations for Producer ID expiration
checks are exposed.

Looking forward to your feedback.

Cheers,
Jorge.

[jira] [Resolved] (KAFKA-16350) StateUpdater does not init transaction after canceling task close action

[Bruno Cadonna (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bruno Cadonna resolved KAFKA-16350.
---
Resolution: Fixed

> StateUpdater does not init transaction after canceling task close action
> 
>
> Key: KAFKA-16350
> URL: https://issues.apache.org/jira/browse/KAFKA-16350
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Matthias J. Sax
>Assignee: Bruno Cadonna
>Priority: Major
> Attachments: 
> tyh5pkfmgwfoe-org.apache.kafka.streams.integration.EosIntegrationTest-shouldWriteLatestOffsetsToCheckpointOnShutdown[exactly_once_v2,
>  processing threads true]-1-output.txt
>
>
> With EOSv2, we use a thread producer shared across all tasks. We init tx on 
> the producer with each _task_ (due to EOSv1 which uses a producer per task), 
> and have a guard in place to only init tx a single time.
> If we hit an error, we close the producer and create a new one, which is 
> still not initialized for transaction. At the same time, with state updater, 
> we schedule a "close task" action on error.
> For each task we get back, we do cancel the "close task" action, to actually 
> keep the task. If this happens for _all_ tasks, we don't have any task in 
> state CRATED at hand, and thus we never init the producer for transactions, 
> because we assume this was already done.
> On the first `send` request, we crash with an IllegalStateException:{{{}{}}}
> {code:java}
> Invalid transition attempted from state UNINITIALIZED to state IN_TRANSACTION 
> {code}
> This bug is exposed via EOSIntegrationTest (logs attached).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Apache Kafka 3.7.1 release

[Justine Olshan]

Hi Igor,

+1 from me. Thanks for taking on the release.

Justine

On Thu, May 16, 2024 at 12:40 AM Chia-Ping Tsai  wrote:

> +1
>
> Our production will be upgraded to your 3.7.1 regardless of how many paper
> works I have to handle :)
>
> Lucas Brutschy  於 2024年5月16日 週四 下午3:25寫道：
>
> > Hi Igor,
> >
> > thanks a lot!
> > +1
> >
> > Lucas
> >
> > On Thu, May 16, 2024 at 5:21 AM Luke Chen  wrote:
> > >
> > > Hi Igor,
> > >
> > > Thanks for volunteering!
> > > +1
> > >
> > > Luke
> > >
> > > On Wed, May 15, 2024 at 11:15 PM Mickael Maison <
> > mickael.mai...@gmail.com>
> > > wrote:
> > >
> > > > Hi Igor,
> > > >
> > > > Thanks for volunteering, +1
> > > >
> > > > Mickael
> > > >
> > > > On Thu, Apr 25, 2024 at 11:09 AM Igor Soarez  wrote:
> > > > >
> > > > > Hi everyone,
> > > > >
> > > > > I'd like to volunteer to be the release manager for a 3.7.1
> release.
> > > > >
> > > > > Please keep in mind, this would be my first release, so I might
> have
> > > > some questions,
> > > > > and it might also take me a bit longer to work through the release
> > > > process.
> > > > > So I'm thinking a good target would be toward the end of May.
> > > > >
> > > > > Please let me know your thoughts and if there are any objections or
> > > > concerns.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > --
> > > > > Igor
> > > >
> >
>

Re: [VOTE] KIP-932: Queues for Kafka

[Omnia Ibrahim]

Thanks for the KIP Andrew +1 none binding from me

> On 16 May 2024, at 14:23, David Jacot  wrote:
> 
> Hi Andrew,
> 
> Thanks for the KIP! This is really exciting! +1 (binding) from me.
> 
> One note regarding the partition assignor interface changes that you
> proposed, it would be great to get the changes in 3.8 in order to not break
> the API of KIP-848 after the preview.
> 
> Best,
> David
> 
> On Wed, May 15, 2024 at 10:37 PM Jun Rao  wrote:
> 
>> Hi, Andrew,
>> 
>> Thanks for the update. Should we mark whether those metrics are
>> standard/required for KIP-714?
>> 
>> Jun
>> 
>> On Tue, May 14, 2024 at 7:31 AM Andrew Schofield <
>> andrew_schofi...@live.com>
>> wrote:
>> 
>>> Hi,
>>> I have made a small update to the KIP as a result of testing the new
>>> share consumer with client telemetry (KIP-714).
>>> 
>>> I’ve added telemetry metric names to the table of client metrics and
>>> also updated the metric group names so that the resulting client metrics
>>> sent to the broker have consistent names.
>>> 
>>> Thanks,
>>> Andrew
>>> 
 On 8 May 2024, at 12:51, Manikumar  wrote:
 
 Hi Andrew,
 
 Thanks for the KIP.  Great write-up!
 
 +1 (binding)
 
 Thanks,
 
 On Wed, May 8, 2024 at 12:17 PM Satish Duggana <
>> satish.dugg...@gmail.com>
>>> wrote:
> 
> Hi Andrew,
> Thanks for the nice KIP, it will allow other messaging use cases to be
> onboarded to Kafka.
> 
> +1 from me.
> 
> Satish.
> 
> On Tue, 7 May 2024 at 03:41, Jun Rao 
>> wrote:
>> 
>> Hi, Andrew,
>> 
>> Thanks for the KIP. +1
>> 
>> Jun
>> 
>> On Mon, Mar 18, 2024 at 11:00 AM Edoardo Comar <
>> edoardli...@gmail.com>
>> wrote:
>> 
>>> Thanks Andrew,
>>> 
>>> +1 (binding)
>>> 
>>> Edo
>>> 
>>> On Mon, 18 Mar 2024 at 16:32, Kenneth Eversole
>>>  wrote:
 
 Hi Andrew
 
 + 1 (Non-Binding)
 
 This will be great addition to Kafka
 
 On Mon, Mar 18, 2024 at 8:27 AM Apoorv Mittal <
>>> apoorvmitta...@gmail.com>
 wrote:
 
> Hi Andrew,
> Thanks for writing the KIP. This is indeed going to be a valuable
>>> addition
> to the Kafka, excited to see the KIP.
> 
> + 1 (Non-Binding)
> 
> Regards,
> Apoorv Mittal
> +44 7721681581
> 
> 
> On Sun, Mar 17, 2024 at 11:16 PM Andrew Schofield <
> andrew_schofield_j...@outlook.com> wrote:
> 
>> Hi,
>> I’ve been working to complete KIP-932 over the past few months
>> and
>> discussions have quietened down.
>> 
>> I’d like to open the voting for KIP-932:
>> 
>> 
>> 
> 
>>> 
>>> 
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka
>> 
>> Thanks,
>> Andrew
> 
>>> 
>>> 
>>> 
>> 

[jira] [Created] (KAFKA-16786) New consumer subscribe should not require the deprecated partition.assignment.strategy

[Lianet Magrans (Jira)]

Lianet Magrans created KAFKA-16786:
--

 Summary: New consumer subscribe should not require the deprecated 
partition.assignment.strategy
 Key: KAFKA-16786
 URL: https://issues.apache.org/jira/browse/KAFKA-16786
 Project: Kafka
  Issue Type: Bug
  Components: consumer
Affects Versions: 3.7.0
Reporter: Lianet Magrans


The partition.assignment.strategy config is deprecated with the new consumer 
group protocol KIP-848. With the new protocol, server side assignors are 
supported for now, defined in the property
group.remote.assignor, and with default values selected by the broker, so it's 
not even a required property. 

The new AsyncKafkaConsumer supports the new protocol only, but it currently 
throws an IllegalStateException if a call to subscribe is made and the 
deprecated config partition.assignment.strategy is empty (see 
[throwIfNoAssignorsConfigured|https://github.com/apache/kafka/blob/056d232f4e28bf8e67e00f8ed2c103fdb0f3b78e/clients/src/main/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumer.java#L1715]).
 
We should remove the reference to ConsumerPartitionAssignor in the 
AsyncKafkaConsumer, along with it's validation for non-empty on subscribe (only 
use it has)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-989: RocksDB Iterator Metrics

[Nick Telford]

Actually, one other point: our existing state store operation metrics are
measured in nanoseconds[1].

Should iterator-duration-(avg|max) also be measured in nanoseconds, for
consistency, or should we keep them milliseconds, as the KIP currently
states?

1:
https://docs.confluent.io/platform/current/streams/monitoring.html#state-store-metrics

On Thu, 16 May 2024 at 12:15, Nick Telford  wrote:

> Good point! I've updated it to "Improved StateStore Iterator metrics for
> detecting leaks" - let me know if you have a better suggestion.
>
> This should affect the voting imo, as nothing of substance has changed.
>
> Regards,
> Nick
>
> On Thu, 16 May 2024 at 01:39, Sophie Blee-Goldman 
> wrote:
>
>> One quick thing -- can you update the title of this KIP to reflect the
>> decision to implement these metrics for all state stores implementations
>> rather than just RocksDB?
>>
>>
>> On Tue, May 14, 2024 at 1:36 PM Nick Telford 
>> wrote:
>>
>> > Woops! Thanks for the catch Lucas. Given this was just a typo, I don't
>> > think this affects the voting.
>> >
>> > Cheers,
>> > Nick
>> >
>> > On Tue, 14 May 2024 at 18:06, Lucas Brutschy > > .invalid>
>> > wrote:
>> >
>> > > Hi Nick,
>> > >
>> > > you are still referring to oldest-open-iterator-age-ms in the
>> > > `Proposed Changes` section.
>> > >
>> > > Cheers,
>> > > Lucas
>> > >
>> > > On Thu, May 2, 2024 at 4:00 PM Lucas Brutschy > >
>> > > wrote:
>> > > >
>> > > > Hi Nick!
>> > > >
>> > > > I agree, the age variant is a bit nicer since the semantics are very
>> > > > clear from the name. If you'd rather go for the simple
>> implementation,
>> > > > how about calling it `oldest-iterator-open-since-ms`? I believe this
>> > > > could be understood without docs. Either way, I think we should be
>> > > > able to open the vote for this KIP because nobody raised any major /
>> > > > blocking concerns.
>> > > >
>> > > > Looking forward to getting this voted on soon!
>> > > >
>> > > > Cheers
>> > > > Lucas
>> > > >
>> > > > On Sun, Mar 31, 2024 at 5:23 PM Nick Telford <
>> nick.telf...@gmail.com>
>> > > wrote:
>> > > > >
>> > > > > Hi Matthias,
>> > > > >
>> > > > > > For the oldest iterator metric, I would propose something simple
>> > like
>> > > > > > `iterator-opened-ms` and it would just be the actual timestamp
>> when
>> > > the
>> > > > > > iterator was opened. I don't think we need to compute the actual
>> > age,
>> > > > > > but user can to this computation themselves?
>> > > > >
>> > > > > That works for me; it's easier to implement like that :-D I'm a
>> > little
>> > > > > concerned that the name "iterator-opened-ms" may not be obvious
>> > enough
>> > > > > without reading the docs.
>> > > > >
>> > > > > > If we think reporting the age instead of just the timestamp is
>> > > better, I
>> > > > > > would propose `iterator-max-age-ms`. I should be sufficient to
>> call
>> > > out
>> > > > > > (as it's kinda "obvious" anyway) that the metric applies to open
>> > > > > > iterator only.
>> > > > >
>> > > > > While I think it's preferable to record the timestamp, rather than
>> > the
>> > > age,
>> > > > > this does have the benefit of a more obvious metric name.
>> > > > >
>> > > > > > Nit: the KIP says it's a store-level metric, but I think it
>> would
>> > be
>> > > > > > good to say explicitly that it's recorded with DEBUG level only?
>> > > > >
>> > > > > Yes, I've already updated the KIP with this information in the
>> table.
>> > > > >
>> > > > > Regards,
>> > > > >
>> > > > > Nick
>> > > > >
>> > > > > On Sun, 31 Mar 2024 at 10:53, Matthias J. Sax 
>> > > wrote:
>> > > > >
>> > > > > > The time window thing was just an idea. Happy to drop it.
>> > > > > >
>> > > > > > For the oldest iterator metric, I would propose something simple
>> > like
>> > > > > > `iterator-opened-ms` and it would just be the actual timestamp
>> when
>> > > the
>> > > > > > iterator was opened. I don't think we need to compute the actual
>> > age,
>> > > > > > but user can to this computation themselves?
>> > > > > >
>> > > > > > If we think reporting the age instead of just the timestamp is
>> > > better, I
>> > > > > > would propose `iterator-max-age-ms`. I should be sufficient to
>> call
>> > > out
>> > > > > > (as it's kinda "obvious" anyway) that the metric applies to open
>> > > > > > iterator only.
>> > > > > >
>> > > > > > And yes, I was hoping that the code inside MetereXxxStore might
>> > > already
>> > > > > > be setup in a way that custom stores would inherit the iterator
>> > > metrics
>> > > > > > automatically -- I am just not sure, and left it as an exercise
>> for
>> > > > > > somebody to confirm :)
>> > > > > >
>> > > > > >
>> > > > > > Nit: the KIP says it's a store-level metric, but I think it
>> would
>> > be
>> > > > > > good to say explicitly that it's recorded with DEBUG level only?
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > -Matthias
>> > > > > >
>> > > > > >
>> > > > > > On 3/28/24 2:52 PM, Nick Telford wrote:
>> > > > > > > Quick addendum:
>> > > 

[jira] [Created] (KAFKA-16785) Migrate TopicBasedRemoteLogMetadataManagerRestartTest to new test infra

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-16785:
--

 Summary: Migrate TopicBasedRemoteLogMetadataManagerRestartTest to 
new test infra
 Key: KAFKA-16785
 URL: https://issues.apache.org/jira/browse/KAFKA-16785
 Project: Kafka
  Issue Type: Test
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai


as title



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16784) Migrate TopicBasedRemoteLogMetadataManagerMultipleSubscriptionsTest to new test infra

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-16784:
--

 Summary: Migrate 
TopicBasedRemoteLogMetadataManagerMultipleSubscriptionsTest to new test infra
 Key: KAFKA-16784
 URL: https://issues.apache.org/jira/browse/KAFKA-16784
 Project: Kafka
  Issue Type: Test
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai


as title



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16783) Migrate RemoteLogMetadataManagerTest to new test infra

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-16783:
--

 Summary: Migrate RemoteLogMetadataManagerTest to new test infra
 Key: KAFKA-16783
 URL: https://issues.apache.org/jira/browse/KAFKA-16783
 Project: Kafka
  Issue Type: Test
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai


as title

`TopicBasedRemoteLogMetadataManagerWrapperWithHarness` could be replaced by 
`RemoteLogMetadataManagerTestUtils#builder`



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-932: Queues for Kafka

[David Jacot]

Hi Andrew,

Thanks for the KIP! This is really exciting! +1 (binding) from me.

One note regarding the partition assignor interface changes that you
proposed, it would be great to get the changes in 3.8 in order to not break
the API of KIP-848 after the preview.

Best,
David

On Wed, May 15, 2024 at 10:37 PM Jun Rao  wrote:

> Hi, Andrew,
>
> Thanks for the update. Should we mark whether those metrics are
> standard/required for KIP-714?
>
> Jun
>
> On Tue, May 14, 2024 at 7:31 AM Andrew Schofield <
> andrew_schofi...@live.com>
> wrote:
>
> > Hi,
> > I have made a small update to the KIP as a result of testing the new
> > share consumer with client telemetry (KIP-714).
> >
> > I’ve added telemetry metric names to the table of client metrics and
> > also updated the metric group names so that the resulting client metrics
> > sent to the broker have consistent names.
> >
> > Thanks,
> > Andrew
> >
> > > On 8 May 2024, at 12:51, Manikumar  wrote:
> > >
> > > Hi Andrew,
> > >
> > > Thanks for the KIP.  Great write-up!
> > >
> > > +1 (binding)
> > >
> > > Thanks,
> > >
> > > On Wed, May 8, 2024 at 12:17 PM Satish Duggana <
> satish.dugg...@gmail.com>
> > wrote:
> > >>
> > >> Hi Andrew,
> > >> Thanks for the nice KIP, it will allow other messaging use cases to be
> > >> onboarded to Kafka.
> > >>
> > >> +1 from me.
> > >>
> > >> Satish.
> > >>
> > >> On Tue, 7 May 2024 at 03:41, Jun Rao 
> wrote:
> > >>>
> > >>> Hi, Andrew,
> > >>>
> > >>> Thanks for the KIP. +1
> > >>>
> > >>> Jun
> > >>>
> > >>> On Mon, Mar 18, 2024 at 11:00 AM Edoardo Comar <
> edoardli...@gmail.com>
> > >>> wrote:
> > >>>
> >  Thanks Andrew,
> > 
> >  +1 (binding)
> > 
> >  Edo
> > 
> >  On Mon, 18 Mar 2024 at 16:32, Kenneth Eversole
> >   wrote:
> > >
> > > Hi Andrew
> > >
> > > + 1 (Non-Binding)
> > >
> > > This will be great addition to Kafka
> > >
> > > On Mon, Mar 18, 2024 at 8:27 AM Apoorv Mittal <
> > apoorvmitta...@gmail.com>
> > > wrote:
> > >
> > >> Hi Andrew,
> > >> Thanks for writing the KIP. This is indeed going to be a valuable
> >  addition
> > >> to the Kafka, excited to see the KIP.
> > >>
> > >> + 1 (Non-Binding)
> > >>
> > >> Regards,
> > >> Apoorv Mittal
> > >> +44 7721681581
> > >>
> > >>
> > >> On Sun, Mar 17, 2024 at 11:16 PM Andrew Schofield <
> > >> andrew_schofield_j...@outlook.com> wrote:
> > >>
> > >>> Hi,
> > >>> I’ve been working to complete KIP-932 over the past few months
> and
> > >>> discussions have quietened down.
> > >>>
> > >>> I’d like to open the voting for KIP-932:
> > >>>
> > >>>
> > >>>
> > >>
> > 
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka
> > >>>
> > >>> Thanks,
> > >>> Andrew
> > >>
> > 
> >
> >
>

please let me subscribe

[Harry Fallows]

please

Harry Fallows

Software Engineer

Email: hfall...@thoughtmachine.net

Web: www.thoughtmachine.net


7 Herbrand Street | London | WC1N 1EX



Data Classification: Internal

-- 
Thought Machine Group Limited, a company registered in England & Wales.
Registered number: 4277. 
Registered Office: 5 New Street Square, 
London EC4A 3TW 
.


The content of this email is confidential and intended for the recipient 
specified in message only. It is strictly forbidden to share any part of 
this message with any third party, without a written consent of the sender. 
If you received this message by mistake, please reply to this message and 
follow with its deletion, so that we can ensure such a mistake does not 
occur in the future.

[jira] [Created] (KAFKA-16782) Some partition's segments are suddenly not deleted anymore

[Roland Sommer (Jira)]

Roland Sommer created KAFKA-16782:
-

 Summary: Some partition's segments are suddenly not deleted anymore
 Key: KAFKA-16782
 URL: https://issues.apache.org/jira/browse/KAFKA-16782
 Project: Kafka
  Issue Type: Bug
Reporter: Roland Sommer


I recently discovered an odd behaviour in one of our kafka clusters 
(KRaft-based, v3.7.0):

We have a topic for distributed log collection with 48 partitions. Retention is 
set to 84 hours, we have the default {{cleanup.policy=delete}} in place. For 
all but two partitions this works as expected. In two partition directories 
there are files going back to january and consuming the specific partitions 
yields data from january (showing it's not only the files lying around, they 
are actually processed).

Topic settings as per {{kafka-topics.sh --describe}}:

{{Topic: syslog TopicId: AeJLnYPnQFOtMc0ZjpH7sw PartitionCount: 48 
ReplicationFactor: 2 Configs: 
compression.type=snappy,cleanup.policy=delete,segment.bytes=1073741824,retention.ms=30240,max.message.bytes=2097152}}

Searching the cluster logs, there is no indicator of what could be the reason 
here (at least I did not spot anything suspicious up until now). Up to the time 
were deletion stopped, there are log entries showing the deleteion of old log 
segments, but that simply stopped. As far as I can see, there has not been any 
change on the cluster at that point.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-989: RocksDB Iterator Metrics

[Nick Telford]

Good point! I've updated it to "Improved StateStore Iterator metrics for
detecting leaks" - let me know if you have a better suggestion.

This should affect the voting imo, as nothing of substance has changed.

Regards,
Nick

On Thu, 16 May 2024 at 01:39, Sophie Blee-Goldman 
wrote:

> One quick thing -- can you update the title of this KIP to reflect the
> decision to implement these metrics for all state stores implementations
> rather than just RocksDB?
>
>
> On Tue, May 14, 2024 at 1:36 PM Nick Telford 
> wrote:
>
> > Woops! Thanks for the catch Lucas. Given this was just a typo, I don't
> > think this affects the voting.
> >
> > Cheers,
> > Nick
> >
> > On Tue, 14 May 2024 at 18:06, Lucas Brutschy  > .invalid>
> > wrote:
> >
> > > Hi Nick,
> > >
> > > you are still referring to oldest-open-iterator-age-ms in the
> > > `Proposed Changes` section.
> > >
> > > Cheers,
> > > Lucas
> > >
> > > On Thu, May 2, 2024 at 4:00 PM Lucas Brutschy 
> > > wrote:
> > > >
> > > > Hi Nick!
> > > >
> > > > I agree, the age variant is a bit nicer since the semantics are very
> > > > clear from the name. If you'd rather go for the simple
> implementation,
> > > > how about calling it `oldest-iterator-open-since-ms`? I believe this
> > > > could be understood without docs. Either way, I think we should be
> > > > able to open the vote for this KIP because nobody raised any major /
> > > > blocking concerns.
> > > >
> > > > Looking forward to getting this voted on soon!
> > > >
> > > > Cheers
> > > > Lucas
> > > >
> > > > On Sun, Mar 31, 2024 at 5:23 PM Nick Telford  >
> > > wrote:
> > > > >
> > > > > Hi Matthias,
> > > > >
> > > > > > For the oldest iterator metric, I would propose something simple
> > like
> > > > > > `iterator-opened-ms` and it would just be the actual timestamp
> when
> > > the
> > > > > > iterator was opened. I don't think we need to compute the actual
> > age,
> > > > > > but user can to this computation themselves?
> > > > >
> > > > > That works for me; it's easier to implement like that :-D I'm a
> > little
> > > > > concerned that the name "iterator-opened-ms" may not be obvious
> > enough
> > > > > without reading the docs.
> > > > >
> > > > > > If we think reporting the age instead of just the timestamp is
> > > better, I
> > > > > > would propose `iterator-max-age-ms`. I should be sufficient to
> call
> > > out
> > > > > > (as it's kinda "obvious" anyway) that the metric applies to open
> > > > > > iterator only.
> > > > >
> > > > > While I think it's preferable to record the timestamp, rather than
> > the
> > > age,
> > > > > this does have the benefit of a more obvious metric name.
> > > > >
> > > > > > Nit: the KIP says it's a store-level metric, but I think it would
> > be
> > > > > > good to say explicitly that it's recorded with DEBUG level only?
> > > > >
> > > > > Yes, I've already updated the KIP with this information in the
> table.
> > > > >
> > > > > Regards,
> > > > >
> > > > > Nick
> > > > >
> > > > > On Sun, 31 Mar 2024 at 10:53, Matthias J. Sax 
> > > wrote:
> > > > >
> > > > > > The time window thing was just an idea. Happy to drop it.
> > > > > >
> > > > > > For the oldest iterator metric, I would propose something simple
> > like
> > > > > > `iterator-opened-ms` and it would just be the actual timestamp
> when
> > > the
> > > > > > iterator was opened. I don't think we need to compute the actual
> > age,
> > > > > > but user can to this computation themselves?
> > > > > >
> > > > > > If we think reporting the age instead of just the timestamp is
> > > better, I
> > > > > > would propose `iterator-max-age-ms`. I should be sufficient to
> call
> > > out
> > > > > > (as it's kinda "obvious" anyway) that the metric applies to open
> > > > > > iterator only.
> > > > > >
> > > > > > And yes, I was hoping that the code inside MetereXxxStore might
> > > already
> > > > > > be setup in a way that custom stores would inherit the iterator
> > > metrics
> > > > > > automatically -- I am just not sure, and left it as an exercise
> for
> > > > > > somebody to confirm :)
> > > > > >
> > > > > >
> > > > > > Nit: the KIP says it's a store-level metric, but I think it would
> > be
> > > > > > good to say explicitly that it's recorded with DEBUG level only?
> > > > > >
> > > > > >
> > > > > >
> > > > > > -Matthias
> > > > > >
> > > > > >
> > > > > > On 3/28/24 2:52 PM, Nick Telford wrote:
> > > > > > > Quick addendum:
> > > > > > >
> > > > > > > My suggested metric "oldest-open-iterator-age-seconds" should
> be
> > > > > > > "oldest-open-iterator-age-ms". Milliseconds is obviously a
> better
> > > > > > > granularity for such a metric.
> > > > > > >
> > > > > > > Still accepting suggestions for a better name.
> > > > > > >
> > > > > > > On Thu, 28 Mar 2024 at 13:41, Nick Telford <
> > nick.telf...@gmail.com
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > >> Hi everyone,
> > > > > > >>
> > > > > > >> Sorry for leaving this for so long. So much for "3 weeks until
> > KIP
> > > > > > freeze"!

Re: [DISCUSS] KIP-1042 support for wildcard when creating new acls

[Claude Warren]

I think that a Trie solution for LITERAL and PREFIX would be faster than
attempting to use the wildcard search strategies on them.  The reasons
being

   - that the wildcard search strategy may return some non-matching (false
   positive) patterns that have to be checked and ignored.
   - the size of the Bloom filters used for searching may be artificially
   inflated because of the length of the LITERAL matches (PREFIX matches too
   but LITERAL will probably be longer anyway)
   - The WILDCARD type will require a Bloom filter and it should be created
   once and accept the 80-100 byte of overhead rather than the calculation
   time on every use.



On Wed, May 15, 2024 at 5:00 PM Murali Basani 
wrote:

> Thank you Haruki for the feedback.
> There are certainly many such use cases where customers ended up creating
> custom authorizers to handle these MATCH based patterns.
> And regarding updating method matchingAcls, Claude updated the KIP
> mentioning an approach, which definitely optimizes the flow, but can be
> discussed further during the PR implementation I believe.
>
> Thank you Greg for the feedback.
> I understand with trie, existing mechanisms for PREFIX/LITERAL can be
> optimized. How about even trying this matchingAcls with a new
> implementation which Claude proposed, may be we can improve the existing
> flows. Probably we can see those details during PR development.
>
> If we don't achieve the expected results as per AuthorizerBenchmark, we can
> drop this kip.
>
> And thank you Claude for the suggestion on the new implementation.
>
> On Tue, May 7, 2024 at 4:37 PM Claude Warren, Jr
>  wrote:
>
> > I have updated KIP-1042 with a proposal for how to reduce the time spent
> > looking for matching wildcard patterns.  Experimentally I see a reduction
> > of 66-75% execution time.
> >
> > On Mon, May 6, 2024 at 9:03 PM Greg Harris  >
> > wrote:
> >
> > > Hi Murali,
> > >
> > > Thanks for the KIP!
> > >
> > > I think I understand the motivation for this KIP in situations where
> > > there are a "cross product" of topics for two or more variables X and
> > > Y, and want to write ACLs for each of the variable axes.
> > > If you format your topics "X-Y-suffix", it's not easy to write rules
> > > that apply to all "Y" topics, because you need to enumerate all of the
> > > "X" values, and the problem persists even if you reorder the topic
> > > name.
> > >
> > > In my recent work on KIP-986 I found it necessary to introduce
> > > "namespaces" to group topics together, and I was going to replicate
> > > the ACL system to specify those namespaces. This change to the ACL
> > > system could increase the expressiveness and complexity of that
> > > feature, if it is ever implemented.
> > > One of the primitives I needed when specifying namespaces was the
> > > ability to tell when two namespaces overlapped (i.e. does there exist
> > > any topic which is present in both namespaces). This is trivial to do
> > > with the current PREFIX and LITERAL system, as we can find the
> > > maximum-length common prefix with just some length comparisons and
> > > equality checks.
> > > I considered specifying namespaces via regular expressions, and found
> > > that it was computationally much more difficult. Computing the
> > > intersection of two regexes appears to be exponential in the length of
> > > the regexes, leading me to avoid adding it.
> > >
> > > I understand that you're not suggesting full REGEX support, and that
> > > "namespaces" don't need to support MATCH, but I think MATCH may run
> > > into related difficulties. Any MATCH can overlap with any other MATCH
> > > or PREFIX if it includes a sufficient number of wildcards. For
> > > example:
> > > MATCH *-accounts-* has overlap with PREFIX nl as they can both match
> > > "nl-accounts-localtopic", but that isn't sensitive to the contents
> > > "nl", it is true for any PREFIX.
> > > MATCH *-accounts-* has overlap with MATCH *localtopic, as they can
> > > both match "nl-accounts-localtopic", but that isn't actually sensitive
> > > to the contents "localtopic", it's true for any MATCH which includes a
> > > wildcard at the beginning.
> > >
> > > This has implications for execution complexity: If we can't compute
> > > whether two patterns overlap, then we need to run both of them on each
> > > piece of input to test if they both match. Under the current
> > > LITERAL/PREFIX system, we can optimize execution with a trie, but that
> > > option wouldn't be available to us with MATCH.
> > >
> > > The current system makes users evaluate a trade-off:
> > > 1. Optimize the number of ACLs by organizing topics according to
> > > prefixes (for example, "accounts-localtopic-nl" and PREFIX "accounts",
> > > PREFIX "accounts-localtopic")
> > > 2. Use less-structured topic names, with a corresponding ACL scheme
> > > that has more individual rules.
> > > The system currently informs users of this tradeoff by making them
> > > write multiple ACLs, and making them think "there 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2908

[Apache Jenkins Server]

See 

[jira] [Resolved] (KAFKA-16668) Enable to set tags by `ClusterTest`

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-16668.

Fix Version/s: 3.8.0
   Resolution: Fixed

> Enable to set tags by `ClusterTest` 
> 
>
> Key: KAFKA-16668
> URL: https://issues.apache.org/jira/browse/KAFKA-16668
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Chia-Ping Tsai
>Assignee: Johnny Hsu
>Priority: Minor
> Fix For: 3.8.0
>
>
> Currently, the display name can be customized by only `name` 
> (https://github.com/apache/kafka/blob/trunk/core/src/test/java/kafka/test/annotation/ClusterTest.java#L42).
>  However, the "key" is hard-code to "name=xxx". Also, it is impossible to set 
> more "tags" for display name. 
> https://github.com/apache/kafka/pull/15766 is a example that we want to add 
> "xxx=bbb" to display name.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-950: Tiered Storage Disablement

[Luke Chen]

Thanks Chia-Ping!
Since ZK is going to be removed, I agree the KRaft part has higher priority.
But if Christo or the community contributor has spare time, it's good to
have ZK part, too!

Thanks.
Luke

On Thu, May 16, 2024 at 5:45 PM Chia-Ping Tsai  wrote:

> +1 but I prefer to ship it to KRaft only.
>
> I do concern that community have enough time to accept more feature in 3.8
> :(
>
> Best,
> Chia-Ping
>
> On 2024/05/14 15:20:50 Christo Lolov wrote:
> > Heya!
> >
> > I would like to start a vote on KIP-950: Tiered Storage Disablement in
> > order to catch the last Kafka release targeting Zookeeper -
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-950%3A++Tiered+Storage+Disablement
> >
> > Best,
> > Christo
> >
>

[jira] [Resolved] (KAFKA-16539) Can't update specific broker configs in pre-migration mode

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-16539.

Resolution: Fixed

> Can't update specific broker configs in pre-migration mode
> --
>
> Key: KAFKA-16539
> URL: https://issues.apache.org/jira/browse/KAFKA-16539
> Project: Kafka
>  Issue Type: Bug
>  Components: config, kraft
>Affects Versions: 3.6.0, 3.7.0, 3.6.1, 3.6.2
>Reporter: David Arthur
>Assignee: David Arthur
>Priority: Major
> Fix For: 3.8.0, 3.7.1
>
>
> In migration mode, ZK brokers will have a forwarding manager configured. This 
> is used to forward requests to the KRaft controller once we get to that part 
> of the migration. However, prior to KRaft taking over as the controller 
> (known as pre-migration mode), the ZK brokers are still attempting to forward 
> IncrementalAlterConfigs to the controller.
> This works fine for cluster level configs (e.g., "-entity-type broker 
> --entity-default"), but this fails for specific broker configs (e.g., 
> "-entity-type broker --entity-id 1").
> This affects BROKER and BROKER_LOGGER config types.
> To workaround this bug, you can either disable migrations on the brokers 
> (assuming no migration has taken place), or proceed with the migration and 
> get to the point where KRaft is the controller.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-950: Tiered Storage Disablement

[Chia-Ping Tsai]

+1 but I prefer to ship it to KRaft only. 

I do concern that community have enough time to accept more feature in 3.8 :(

Best,
Chia-Ping

On 2024/05/14 15:20:50 Christo Lolov wrote:
> Heya!
> 
> I would like to start a vote on KIP-950: Tiered Storage Disablement in
> order to catch the last Kafka release targeting Zookeeper -
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-950%3A++Tiered+Storage+Disablement
> 
> Best,
> Christo
> 

[jira] [Created] (KAFKA-16781) Expose advertised.listeners in controller node

[Luke Chen (Jira)]

Luke Chen created KAFKA-16781:
-

 Summary: Expose advertised.listeners in controller node
 Key: KAFKA-16781
 URL: https://issues.apache.org/jira/browse/KAFKA-16781
 Project: Kafka
  Issue Type: Improvement
Reporter: Luke Chen


After 
[KIP-919|https://cwiki.apache.org/confluence/display/KAFKA/KIP-919%3A+Allow+AdminClient+to+Talk+Directly+with+the+KRaft+Controller+Quorum+and+add+Controller+Registration],
 we allow clients to talk to the KRaft controller node directly. But unlike 
broker node, we don't allow users to config advertised.listeners for clients to 
connect to. Without this config, the client cannot connect to the controller 
node if the controller is sitting behind NAT network while the client is in the 
external network.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[DISCUSS] KIP-1045 Decide MockAdminClient to move to public api or not

[Muralidhar Basani]

Hello,

As part of this KIP, I would like to take your opinions in
moving MockAdminClient to the src folder, making it public, would it be
beneficial or not.

KIP-1045


Currently MockConsumer and MockProducer are part of the public API. They
are useful for developers wanting to test their applications. On the other
hand MockAdminClient is not part of the public API (it's under test). We
should consider moving it to src so users can also easily test applications
that depend on Admin. (Mentioned by Mickael Maison in KAFKA-15258
)

Thanks,
Murali

Muralidhar Basani
Staff Software Engineer, Aiven
muralidhar.bas...@aiven.io

[jira] [Created] (KAFKA-16780) Txn consumer exerts pressure on remote storage when reading non-txn topic

[Kamal Chandraprakash (Jira)]

Kamal Chandraprakash created KAFKA-16780:


 Summary: Txn consumer exerts pressure on remote storage when 
reading non-txn topic
 Key: KAFKA-16780
 URL: https://issues.apache.org/jira/browse/KAFKA-16780
 Project: Kafka
  Issue Type: Task
Reporter: Kamal Chandraprakash


h3. Logic to read aborted txns:
 # When the consumer enables isolation_level as {{READ_COMMITTED}} and reads a 
non-txn topic, then the broker has to 
[traverse|https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/log/LocalLog.scala#L394]
 all the local log segments to collect the aborted transactions since there 
won't be any entry in the transaction index.
 # The same 
[logic|https://github.com/apache/kafka/blob/trunk/core/src/main/java/kafka/log/remote/RemoteLogManager.java#L1436]
 is applied while reading from remote storage. In this case, when the FETCH 
request is reading data from the first remote log segment, then it has to fetch 
the transaction indexes of all the remaining remote-log segments, and then the 
call lands to the local-log segments before responding to the FETCH request 
which increases the time taken to serve the requests.

The [EoS Abort 
Index|https://docs.google.com/document/d/1Rlqizmk7QCDe8qAnVW5e5X8rGvn6m2DCR3JR2yqwVjc]
 design doc explains how the transaction index file filters out the aborted 
transaction records.

The issue is when consumers are enabled with the {{READ_COMMITTED}} isolation 
level but read the normal topics. If the topic is enabled with the transaction, 
then we expect the transaction to either commit/rollback within 15 minutes 
(default transaction.max.timeout.ms = 15 mins), possibly we may have to search 
only for one (or) two remote log segments.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Apache Kafka 3.7.1 release

[Chia-Ping Tsai]

+1

Our production will be upgraded to your 3.7.1 regardless of how many paper
works I have to handle :)

Lucas Brutschy  於 2024年5月16日 週四 下午3:25寫道：

> Hi Igor,
>
> thanks a lot!
> +1
>
> Lucas
>
> On Thu, May 16, 2024 at 5:21 AM Luke Chen  wrote:
> >
> > Hi Igor,
> >
> > Thanks for volunteering!
> > +1
> >
> > Luke
> >
> > On Wed, May 15, 2024 at 11:15 PM Mickael Maison <
> mickael.mai...@gmail.com>
> > wrote:
> >
> > > Hi Igor,
> > >
> > > Thanks for volunteering, +1
> > >
> > > Mickael
> > >
> > > On Thu, Apr 25, 2024 at 11:09 AM Igor Soarez  wrote:
> > > >
> > > > Hi everyone,
> > > >
> > > > I'd like to volunteer to be the release manager for a 3.7.1 release.
> > > >
> > > > Please keep in mind, this would be my first release, so I might have
> > > some questions,
> > > > and it might also take me a bit longer to work through the release
> > > process.
> > > > So I'm thinking a good target would be toward the end of May.
> > > >
> > > > Please let me know your thoughts and if there are any objections or
> > > concerns.
> > > >
> > > > Thanks,
> > > >
> > > > --
> > > > Igor
> > >
>

Re: [DISCUSS] Apache Kafka 3.7.1 release

[Lucas Brutschy]

Hi Igor,

thanks a lot!
+1

Lucas

On Thu, May 16, 2024 at 5:21 AM Luke Chen  wrote:
>
> Hi Igor,
>
> Thanks for volunteering!
> +1
>
> Luke
>
> On Wed, May 15, 2024 at 11:15 PM Mickael Maison 
> wrote:
>
> > Hi Igor,
> >
> > Thanks for volunteering, +1
> >
> > Mickael
> >
> > On Thu, Apr 25, 2024 at 11:09 AM Igor Soarez  wrote:
> > >
> > > Hi everyone,
> > >
> > > I'd like to volunteer to be the release manager for a 3.7.1 release.
> > >
> > > Please keep in mind, this would be my first release, so I might have
> > some questions,
> > > and it might also take me a bit longer to work through the release
> > process.
> > > So I'm thinking a good target would be toward the end of May.
> > >
> > > Please let me know your thoughts and if there are any objections or
> > concerns.
> > >
> > > Thanks,
> > >
> > > --
> > > Igor
> >

Re: [DISCUSS] Apache Kafka 3.8.0 release

[Josep Prat]

Hi all,

We are now officially past the KIP freeze deadline. KIPs that are approved
after this point in time shouldn't be adopted in the 3.8.x release (except
the 2 already mentioned KIPS 989 and 1028 assuming no vetoes occur).

Reminder of the upcoming deadlines:
- Feature freeze is on May 29th
- Code freeze is June 12th

If you have an approved KIP that you know already you won't be able to
complete before the feature freeze deadline, please update the Release
column in the
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
page.

Thanks all,

On Wed, May 15, 2024 at 8:53 PM Josep Prat  wrote:

> Hi Nick,
>
> If nobody comes up with concerns or pushback until the time of closing the
> vote, I think we can take it for 3.8.
>
> Best,
>
> -
>
> Josep Prat
> Open Source Engineering Director, aivenjosep.p...@aiven.io   |
> +491715557497 | aiven.io
> Aiven Deutschland GmbH
> Alexanderufer 3-7, 10117 Berlin
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> Amtsgericht Charlottenburg, HRB 209739 B
>
> On Wed, May 15, 2024, 20:48 Nick Telford  wrote:
>
>> Hi Josep,
>>
>> Would it be possible to sneak KIP-989 into 3.8? Just as with 1028, it's
>> currently being voted on and has already received the requisite votes. The
>> only thing holding it back is the 72 hour voting window.
>>
>> Vote thread here:
>> https://lists.apache.org/thread/nhr65h4784z49jbsyt5nx8ys81q90k6s
>>
>> Regards,
>>
>> Nick
>>
>> On Wed, 15 May 2024 at 17:47, Josep Prat 
>> wrote:
>>
>> > And my maths are wrong! I added 24 hours more to all the numbers in
>> there.
>> > If after 72 hours no vetoes appear, I have no objections on adding this
>> > specific KIP as it shouldn't have a big blast radius of affectation.
>> >
>> > Best,
>> >
>> > On Wed, May 15, 2024 at 6:44 PM Josep Prat  wrote:
>> >
>> > > Ah, I see Chris was faster writing this than me.
>> > >
>> > > On Wed, May 15, 2024 at 6:43 PM Josep Prat 
>> wrote:
>> > >
>> > >> Hi all,
>> > >> You still have the full day of today (independently for the
>> timezone) to
>> > >> get KIPs approved. Tomorrow morning (CEST timezone) I'll send another
>> > email
>> > >> asking developers to assign future approved KIPs to another version
>> > that is
>> > >> not 3.8.
>> > >>
>> > >> So, the only problem I see with KIP-1028 is that it hasn't been open
>> for
>> > >> a vote for 72 hours (48 hours as of now). If there is no negative
>> > voting on
>> > >> the KIP I think we can let that one in, given it would only miss the
>> > >> deadline by less than 12 hours (if my timezone maths add up).
>> > >>
>> > >> Best,
>> > >>
>> > >> On Wed, May 15, 2024 at 6:35 PM Ismael Juma 
>> wrote:
>> > >>
>> > >>> The KIP freeze is just about having the KIP accepted. Not sure why
>> we
>> > >>> would
>> > >>> need an exception for that.
>> > >>>
>> > >>> Ismael
>> > >>>
>> > >>> On Wed, May 15, 2024 at 9:20 AM Chris Egerton <
>> fearthecel...@gmail.com
>> > >
>> > >>> wrote:
>> > >>>
>> > >>> > FWIW I think that the low blast radius for KIP-1028 should allow
>> it
>> > to
>> > >>> > proceed without adhering to the usual KIP and feature freeze
>> dates.
>> > >>> Code
>> > >>> > freeze is probably worth still  respecting, at least if changes
>> are
>> > >>> > required to the docker/jvm/Dockerfile. But I defer to Josep's
>> > >>> judgement as
>> > >>> > the release manager.
>> > >>> >
>> > >>> > On Wed, May 15, 2024, 06:59 Vedarth Sharma <
>> vedarth.sha...@gmail.com
>> > >
>> > >>> > wrote:
>> > >>> >
>> > >>> > > Hey Josep!
>> > >>> > >
>> > >>> > > The KIP 1028 has received the required votes. Voting thread:-
>> > >>> > >
>> https://lists.apache.org/thread/cdq4wfv5v1gpqlxnf46ycwtcwk5wos4q
>> > >>> > > But we are keeping the vote open for 72 hours as per the
>> process.
>> > >>> > >
>> > >>> > > I would like to request you to please consider it for the 3.8.0
>> > >>> release.
>> > >>> > >
>> > >>> > > Thanks and regards,
>> > >>> > > Vedarth
>> > >>> > >
>> > >>> > >
>> > >>> > > On Wed, May 15, 2024 at 1:14 PM Josep Prat
>> > >>> 
>> > >>> > > wrote:
>> > >>> > >
>> > >>> > > > Hi Kafka developers!
>> > >>> > > >
>> > >>> > > > Today is the KIP freeze deadline. All KIPs should be accepted
>> by
>> > >>> EOD
>> > >>> > > today.
>> > >>> > > > Tomorrow morning (CEST timezone) I'll start summarizing all
>> KIPs
>> > >>> that
>> > >>> > > have
>> > >>> > > > been approved. Please any KIP approved after tomorrow should
>> be
>> > >>> adopted
>> > >>> > > in
>> > >>> > > > a future release version, not 3.8.
>> > >>> > > >
>> > >>> > > > Other relevant upcoming deadlines:
>> > >>> > > > - Feature freeze is on May 29th
>> > >>> > > > - Code freeze is June 12th
>> > >>> > > >
>> > >>> > > > Best,
>> > >>> > > >
>> > >>> > > > On Fri, May 3, 2024 at 3:59 PM Josep Prat <
>> josep.p...@aiven.io>
>> > >>> wrote:
>> > >>> > > >
>> > >>> > > > > Hi Kafka developers!
>> > >>> > > > > I just wanted to remind you all of the upcoming relevant
>> dates
>> > >>> for
>> > >>> > > Kafka
>> >