Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2299

[Apache Jenkins Server]

See 

Re: [PR] MINOR: add docs for tiered storage quick start [kafka-site]

[via GitHub]

showuon commented on PR #562:
URL: https://github.com/apache/kafka-site/pull/562#issuecomment-1767477349

   @satishd  , please take a look when available. Thanks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [DISCUSS] KIP-978: Allow dynamic reloading of certificates with different DN / SANs

[Jakub Scholz]

Hi Viktor,

Thanks for reviewing the KIP and for your comment. I agree that it makes
sense to split them. I do not have any use-case where I would use only one
or the other right now. But it is easier to enable two options now than to
somehow split the option into two later. I updated the KIP accordingly.

Thanks & Regards
Jakub

On Tue, Oct 17, 2023 at 2:44 PM Viktor Somogyi-Vass
 wrote:

> Hi Jakub,
>
> I think the KIP looks good overall, and I have one question for now.
> Would it make sense to split the config you want to introduce
> (ssl.allow.dn.and.san.changes) into two configs? Would users want to enable
> one but not the other?
>
> Thanks,
> Viktor
>
> On Wed, Sep 13, 2023 at 10:00 PM Jakub Scholz  wrote:
>
> > Hi all,
> >
> > I would like to start the discussion about the KIP-978: Allow dynamic
> > reloading of certificates with different DN / SANs
> > <
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263429128
> > >.
> > It proposes adding an option to disable the current validation of the DN
> > and SANs when dynamically changing the keystore. Please have a look and
> let
> > me know your thoughts ...
> >
> > Thanks & Regards
> > Jakub
> >
>

Re: [VOTE] KIP-988 Streams StandbyUpdateListener

[Guozhang Wang]

+1 from me.

On Mon, Oct 16, 2023 at 1:56 AM Lucas Brutschy
 wrote:
>
> Hi,
>
> thanks again for the KIP!
>
> +1 (binding)
>
> Cheers,
> Lucas
>
>
>
> On Sun, Oct 15, 2023 at 9:13 AM Colt McNealy  wrote:
> >
> > Hello there,
> >
> > I'd like to call a vote on KIP-988 (co-authored by my friend and colleague
> > Eduwer Camacaro). We are hoping to get it in before the 3.7.0 release.
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-988%3A+Streams+Standby+Task+Update+Listener
> >
> > Cheers,
> > Colt McNealy
> >
> > *Founder, LittleHorse.dev*

Re: [VOTE] KIP-985 Add reverseRange and reverseAll query over kv-store in IQv2

[Guozhang Wang]

Seems my previous msg was sent to the wrong recipient, just resending..

On Fri, Oct 13, 2023 at 7:06 PM Guozhang Wang
 wrote:
>
> Thanks Hanyu. I made a pass on the KIP and read through the DISCUSS
> thread. Do not have any comments. +1
>
> On Fri, Oct 13, 2023 at 9:29 AM Hanyu (Peter) Zheng
>  wrote:
> >
> > Hello everyone,
> >
> > I would like to start a vote for KIP-985 that Add reverseRange and
> > reverseAll query over kv-store in IQv2.
> >
> > Sincerely,
> > Hanyu
> >
> > On Fri, Oct 13, 2023 at 9:15 AM Hanyu (Peter) Zheng 
> > wrote:
> >
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-985:+Add+reverseRange+and+reverseAll+query+over+kv-store+in+IQv2
> > >
> > > --
> > >
> > > [image: Confluent] 
> > > Hanyu (Peter) Zheng he/him/his
> > > Software Engineer Intern
> > > +1 (213) 431-7193 <+1+(213)+431-7193>
> > > Follow us: [image: Blog]
> > > [image:
> > > Twitter] [image: LinkedIn]
> > > [image: Slack]
> > > [image: YouTube]
> > > 
> > >
> > > [image: Try Confluent Cloud for Free]
> > > 
> > >
> >
> >
> > --
> >
> > [image: Confluent] 
> > Hanyu (Peter) Zheng he/him/his
> > Software Engineer Intern
> > +1 (213) 431-7193 <+1+(213)+431-7193>
> > Follow us: [image: Blog]
> > [image:
> > Twitter] [image: LinkedIn]
> > [image: Slack]
> > [image: YouTube]
> > 
> >
> > [image: Try Confluent Cloud for Free]
> > 

Re: [VOTE] KIP-985 Add reverseRange and reverseAll query over kv-store in IQv2

[Hao Li]

Seems late. But +1 (non-binding).

Thanks for the KIP!

Hao

On Tue, Oct 17, 2023 at 10:27 AM Hanyu (Peter) Zheng
 wrote:

> Hi all,
>
> This voting thread has been open for over 72 hours and has received enough
> votes. Therefore, the vote will be closed now.
>
> +3 binding votes
>
> KIP-985 has PASSED.
>
>
> Thanks all for your votes
> Hanyu
>
> On Tue, Oct 17, 2023 at 8:25 AM Walker Carlson
>  wrote:
>
> > +1 (binding_
> >
> > Thanks!
> >
> > On Tue, Oct 17, 2023 at 3:22 AM Lucas Brutschy
> >  wrote:
> >
> > > +1 (binding)
> > >
> > > Thanks for the KIP!
> > >
> > > On Tue, Oct 17, 2023 at 2:31 AM Matthias J. Sax 
> > wrote:
> > > >
> > > > +1 (binding)
> > > >
> > > >
> > > > On 10/13/23 9:24 AM, Hanyu (Peter) Zheng wrote:
> > > > > Hello everyone,
> > > > >
> > > > > I would like to start a vote for KIP-985 that Add reverseRange and
> > > > > reverseAll query over kv-store in IQv2.
> > > > >
> > > > > Sincerely,
> > > > > Hanyu
> > > > >
> > > > > On Fri, Oct 13, 2023 at 9:15 AM Hanyu (Peter) Zheng <
> > > pzh...@confluent.io>
> > > > > wrote:
> > > > >
> > > > >>
> > > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-985:+Add+reverseRange+and+reverseAll+query+over+kv-store+in+IQv2
> > > > >>
> > > > >> --
> > > > >>
> > > > >> [image: Confluent] 
> > > > >> Hanyu (Peter) Zheng he/him/his
> > > > >> Software Engineer Intern
> > > > >> +1 (213) 431-7193 <+1+(213)+431-7193>
> > > > >> Follow us: [image: Blog]
> > > > >> <
> > >
> >
> https://www.confluent.io/blog?utm_source=footer_medium=email_campaign=ch.email-signature_type.community_content.blog
> > > >[image:
> > > > >> Twitter] [image: LinkedIn]
> > > > >> [image: Slack]
> > > > >> [image: YouTube]
> > > > >> 
> > > > >>
> > > > >> [image: Try Confluent Cloud for Free]
> > > > >> <
> > >
> >
> https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound_source=gmail_medium=organic
> > > >
> > > > >>
> > > > >
> > > > >
> > >
> >
>
>
> --
>
> [image: Confluent] 
> Hanyu (Peter) Zheng he/him/his
> Software Engineer Intern
> +1 (213) 431-7193 <+1+(213)+431-7193>
> Follow us: [image: Blog]
> <
> https://www.confluent.io/blog?utm_source=footer_medium=email_campaign=ch.email-signature_type.community_content.blog
> >[image:
> Twitter] [image: LinkedIn]
> [image: Slack]
> [image: YouTube]
> 
>
> [image: Try Confluent Cloud for Free]
> <
> https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound_source=gmail_medium=organic
> >
>

[jira] [Created] (KAFKA-15628) Refactor ConsumerRebalanceListener invocation for reuse

[Kirk True (Jira)]

Kirk True created KAFKA-15628:
-

 Summary: Refactor ConsumerRebalanceListener invocation for reuse
 Key: KAFKA-15628
 URL: https://issues.apache.org/jira/browse/KAFKA-15628
 Project: Kafka
  Issue Type: Sub-task
  Components: clients, consumer
Reporter: Kirk True
Assignee: Kirk True


Pull out the code related to invoking ConsumerRebalanceListener methods into 
its own class so that it can be reused by the KIP-848 implementation



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Apache Kafka 3.5.2 release

[Matthias J. Sax]

Thanks -- there is a few fixed for Kafka Streams we are considering to 
cherry-pick to get into 3.5.2 release -- what timeline do you target for 
the release?



-Matthias

On 10/17/23 8:47 AM, Divij Vaidya wrote:

Thank you for volunteering Luke.

--
Divij Vaidya



On Tue, Oct 17, 2023 at 3:26 PM Bill Bejeck  wrote:


Thanks for driving the release, Luke.

+1
-Bill

On Tue, Oct 17, 2023 at 5:05 AM Satish Duggana 
wrote:


Thanks Luke for volunteering for 3.5.2 release.

On Tue, 17 Oct 2023 at 11:58, Josep Prat 
wrote:


Hi Luke,

Thanks for taking this one!

Best,

On Tue, Oct 17, 2023 at 8:12 AM Luke Chen  wrote:


Hi all,

I'd like to volunteer as release manager for the Apache Kafka 3.5.2,

to

have an important bug/vulnerability fix release for 3.5.1.

If there are no objections, I'll start building a release plan in

thewiki

in the next couple of weeks.

Thanks,
Luke




--
[image: Aiven] 

*Josep Prat*
Open Source Engineering Director, *Aiven*
josep.p...@aiven.io   |   +491715557497
aiven.io    |   <

https://www.facebook.com/aivencloud>

      <

https://twitter.com/aiven_io>

*Aiven Deutschland GmbH*
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
Amtsgericht Charlottenburg, HRB 209739 B

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2298

[Apache Jenkins Server]

See 

Re: [VOTE] KIP-985 Add reverseRange and reverseAll query over kv-store in IQv2

[Hanyu (Peter) Zheng]

Hi all,

This voting thread has been open for over 72 hours and has received enough
votes. Therefore, the vote will be closed now.

+3 binding votes

KIP-985 has PASSED.


Thanks all for your votes
Hanyu

On Tue, Oct 17, 2023 at 8:25 AM Walker Carlson
 wrote:

> +1 (binding_
>
> Thanks!
>
> On Tue, Oct 17, 2023 at 3:22 AM Lucas Brutschy
>  wrote:
>
> > +1 (binding)
> >
> > Thanks for the KIP!
> >
> > On Tue, Oct 17, 2023 at 2:31 AM Matthias J. Sax 
> wrote:
> > >
> > > +1 (binding)
> > >
> > >
> > > On 10/13/23 9:24 AM, Hanyu (Peter) Zheng wrote:
> > > > Hello everyone,
> > > >
> > > > I would like to start a vote for KIP-985 that Add reverseRange and
> > > > reverseAll query over kv-store in IQv2.
> > > >
> > > > Sincerely,
> > > > Hanyu
> > > >
> > > > On Fri, Oct 13, 2023 at 9:15 AM Hanyu (Peter) Zheng <
> > pzh...@confluent.io>
> > > > wrote:
> > > >
> > > >>
> > > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-985:+Add+reverseRange+and+reverseAll+query+over+kv-store+in+IQv2
> > > >>
> > > >> --
> > > >>
> > > >> [image: Confluent] 
> > > >> Hanyu (Peter) Zheng he/him/his
> > > >> Software Engineer Intern
> > > >> +1 (213) 431-7193 <+1+(213)+431-7193>
> > > >> Follow us: [image: Blog]
> > > >> <
> >
> https://www.confluent.io/blog?utm_source=footer_medium=email_campaign=ch.email-signature_type.community_content.blog
> > >[image:
> > > >> Twitter] [image: LinkedIn]
> > > >> [image: Slack]
> > > >> [image: YouTube]
> > > >> 
> > > >>
> > > >> [image: Try Confluent Cloud for Free]
> > > >> <
> >
> https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound_source=gmail_medium=organic
> > >
> > > >>
> > > >
> > > >
> >
>


-- 

[image: Confluent] 
Hanyu (Peter) Zheng he/him/his
Software Engineer Intern
+1 (213) 431-7193 <+1+(213)+431-7193>
Follow us: [image: Blog]
[image:
Twitter] [image: LinkedIn]
[image: Slack]
[image: YouTube]


[image: Try Confluent Cloud for Free]

[jira] [Created] (KAFKA-15627) KIP-951, Java client changes to incorporate the leader discovery optimisations

[Mayank Shekhar Narula (Jira)]

Mayank Shekhar Narula created KAFKA-15627:
-

 Summary: KIP-951, Java client changes to incorporate the leader 
discovery optimisations
 Key: KAFKA-15627
 URL: https://issues.apache.org/jira/browse/KAFKA-15627
 Project: Kafka
  Issue Type: Improvement
  Components: clients
Affects Versions: 3.7.0
Reporter: Mayank Shekhar Narula
Assignee: Mayank Shekhar Narula


Java client changes for 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-951%3A+Leader+discovery+optimisations+for+the+client



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Apache Kafka 3.5.2 release

[Divij Vaidya]

Thank you for volunteering Luke.

--
Divij Vaidya



On Tue, Oct 17, 2023 at 3:26 PM Bill Bejeck  wrote:

> Thanks for driving the release, Luke.
>
> +1
> -Bill
>
> On Tue, Oct 17, 2023 at 5:05 AM Satish Duggana 
> wrote:
>
> > Thanks Luke for volunteering for 3.5.2 release.
> >
> > On Tue, 17 Oct 2023 at 11:58, Josep Prat 
> > wrote:
> > >
> > > Hi Luke,
> > >
> > > Thanks for taking this one!
> > >
> > > Best,
> > >
> > > On Tue, Oct 17, 2023 at 8:12 AM Luke Chen  wrote:
> > >
> > > > Hi all,
> > > >
> > > > I'd like to volunteer as release manager for the Apache Kafka 3.5.2,
> to
> > > > have an important bug/vulnerability fix release for 3.5.1.
> > > >
> > > > If there are no objections, I'll start building a release plan in
> > thewiki
> > > > in the next couple of weeks.
> > > >
> > > > Thanks,
> > > > Luke
> > > >
> > >
> > >
> > > --
> > > [image: Aiven] 
> > >
> > > *Josep Prat*
> > > Open Source Engineering Director, *Aiven*
> > > josep.p...@aiven.io   |   +491715557497
> > > aiven.io    |   <
> > https://www.facebook.com/aivencloud>
> > >      <
> > https://twitter.com/aiven_io>
> > > *Aiven Deutschland GmbH*
> > > Alexanderufer 3-7, 10117 Berlin
> > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > Amtsgericht Charlottenburg, HRB 209739 B
> >
>

[jira] [Created] (KAFKA-15626) Replace verification guard object with an specific type

[Justine Olshan (Jira)]

Justine Olshan created KAFKA-15626:
--

 Summary: Replace verification guard object with an specific type
 Key: KAFKA-15626
 URL: https://issues.apache.org/jira/browse/KAFKA-15626
 Project: Kafka
  Issue Type: Sub-task
Reporter: Justine Olshan
Assignee: Justine Olshan


 https://github.com/apache/kafka/pull/13787#discussion_r1361468169



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-985 Add reverseRange and reverseAll query over kv-store in IQv2

[Walker Carlson]

+1 (binding_

Thanks!

On Tue, Oct 17, 2023 at 3:22 AM Lucas Brutschy
 wrote:

> +1 (binding)
>
> Thanks for the KIP!
>
> On Tue, Oct 17, 2023 at 2:31 AM Matthias J. Sax  wrote:
> >
> > +1 (binding)
> >
> >
> > On 10/13/23 9:24 AM, Hanyu (Peter) Zheng wrote:
> > > Hello everyone,
> > >
> > > I would like to start a vote for KIP-985 that Add reverseRange and
> > > reverseAll query over kv-store in IQv2.
> > >
> > > Sincerely,
> > > Hanyu
> > >
> > > On Fri, Oct 13, 2023 at 9:15 AM Hanyu (Peter) Zheng <
> pzh...@confluent.io>
> > > wrote:
> > >
> > >>
> > >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-985:+Add+reverseRange+and+reverseAll+query+over+kv-store+in+IQv2
> > >>
> > >> --
> > >>
> > >> [image: Confluent] 
> > >> Hanyu (Peter) Zheng he/him/his
> > >> Software Engineer Intern
> > >> +1 (213) 431-7193 <+1+(213)+431-7193>
> > >> Follow us: [image: Blog]
> > >> <
> https://www.confluent.io/blog?utm_source=footer_medium=email_campaign=ch.email-signature_type.community_content.blog
> >[image:
> > >> Twitter] [image: LinkedIn]
> > >> [image: Slack]
> > >> [image: YouTube]
> > >> 
> > >>
> > >> [image: Try Confluent Cloud for Free]
> > >> <
> https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound_source=gmail_medium=organic
> >
> > >>
> > >
> > >
>

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2297

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-15625) Do not flush global state store at each commit

[Bruno Cadonna (Jira)]

Bruno Cadonna created KAFKA-15625:
-

 Summary: Do not flush global state store at each commit
 Key: KAFKA-15625
 URL: https://issues.apache.org/jira/browse/KAFKA-15625
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Reporter: Bruno Cadonna


Global state stores are flushed at each commit. While that is not a big issue 
with at-least-once processing mode since the commit interval is by default 30s, 
it might become an issue with EOS where the commit interval is 200ms by default.
One option would be to flush and checkpoint global state stores when the delta 
of the content exceeds a given threshold as we do for other stores. See 
https://github.com/apache/kafka/blob/a1f3c6d16061566a4f53c72a95e2679b8ee229e0/streams/src/main/java/org/apache/kafka/streams/processor/internals/AbstractTask.java#L97
 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15624) Reconsider synchronisation of methods in RocksDBStore

[Bruno Cadonna (Jira)]

Bruno Cadonna created KAFKA-15624:
-

 Summary: Reconsider synchronisation of methods in RocksDBStore
 Key: KAFKA-15624
 URL: https://issues.apache.org/jira/browse/KAFKA-15624
 Project: Kafka
  Issue Type: Improvement
Reporter: Bruno Cadonna


The code in {{RocksDBStore}} evolved over time. We should reconsider the 
synchronization of the methods in RocksDBStore. Maybe some synchronizations are 
not needed anymore or can be improved. 
The synchronization of the methods is inconsistent. For example, {{putAll()}} 
is not synchronized whereas {{put()}} is synchronized. That could be because 
once {{putAll()}} was a loop over multiple calls to {{put()}}. Additionally, we 
should reconsider how we validate whether the store is open since that seems to 
be the main reason why we synchronize methods.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15570) Add unit tests for MemoryConfigBackingStore

[Chris Egerton (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Egerton resolved KAFKA-15570.
---
Resolution: Done

> Add unit tests for MemoryConfigBackingStore
> ---
>
> Key: KAFKA-15570
> URL: https://issues.apache.org/jira/browse/KAFKA-15570
> Project: Kafka
>  Issue Type: Test
>  Components: connect, KafkaConnect
>Reporter: Yash Mayya
>Assignee: Yash Mayya
>Priority: Minor
>
> Currently, the 
> [MemoryConfigBackingStore|https://github.com/apache/kafka/blob/6e164bb9ace3ea7a1a9542904d1a01c9fd3a1b48/connect/runtime/src/main/java/org/apache/kafka/connect/storage/MemoryConfigBackingStore.java#L37]
>  class doesn't have any unit tests for its functionality. While most of its 
> functionality is fairly lightweight today, changes will be introduced with 
> [KIP-980|https://cwiki.apache.org/confluence/display/KAFKA/KIP-980%3A+Allow+creating+connectors+in+a+stopped+state]
>  (potentially 
> [KIP-976|https://cwiki.apache.org/confluence/display/KAFKA/KIP-976%3A+Cluster-wide+dynamic+log+adjustment+for+Kafka+Connect]
>  as well) and it would be good to have a test setup in place before those 
> changes are made.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Reopened] (KAFKA-15570) Add unit tests for MemoryConfigBackingStore

[Chris Egerton (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Egerton reopened KAFKA-15570:
---

> Add unit tests for MemoryConfigBackingStore
> ---
>
> Key: KAFKA-15570
> URL: https://issues.apache.org/jira/browse/KAFKA-15570
> Project: Kafka
>  Issue Type: Test
>  Components: connect, KafkaConnect
>Reporter: Yash Mayya
>Assignee: Yash Mayya
>Priority: Minor
>
> Currently, the 
> [MemoryConfigBackingStore|https://github.com/apache/kafka/blob/6e164bb9ace3ea7a1a9542904d1a01c9fd3a1b48/connect/runtime/src/main/java/org/apache/kafka/connect/storage/MemoryConfigBackingStore.java#L37]
>  class doesn't have any unit tests for its functionality. While most of its 
> functionality is fairly lightweight today, changes will be introduced with 
> [KIP-980|https://cwiki.apache.org/confluence/display/KAFKA/KIP-980%3A+Allow+creating+connectors+in+a+stopped+state]
>  (potentially 
> [KIP-976|https://cwiki.apache.org/confluence/display/KAFKA/KIP-976%3A+Cluster-wide+dynamic+log+adjustment+for+Kafka+Connect]
>  as well) and it would be good to have a test setup in place before those 
> changes are made.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15623) Migrate remaining tests in streams module to JUnit 5

[Ismael Juma (Jira)]

Ismael Juma created KAFKA-15623:
---

 Summary: Migrate remaining tests in streams module to JUnit 5
 Key: KAFKA-15623
 URL: https://issues.apache.org/jira/browse/KAFKA-15623
 Project: Kafka
  Issue Type: Sub-task
Reporter: Ismael Juma






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-12220) Replace PowerMock by Mockito

[Ismael Juma (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-12220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-12220.
-
Resolution: Fixed

> Replace PowerMock by Mockito
> 
>
> Key: KAFKA-12220
> URL: https://issues.apache.org/jira/browse/KAFKA-12220
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Major
>
> We are migrating project from junit 4 to junit 5 (KAFKA-7339). PowerMock, 
> however, does not support junit 5 totally 
> (https://github.com/powermock/powermock/issues/830). Hence, we ought to 
> replace PowerMock by Mockito before migrating to junit 5 since rewriting all 
> tests which are depending on PowerMock can bring a bunch of changes.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Reopened] (KAFKA-12220) Replace PowerMock by Mockito

[Ismael Juma (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-12220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma reopened KAFKA-12220:
-

> Replace PowerMock by Mockito
> 
>
> Key: KAFKA-12220
> URL: https://issues.apache.org/jira/browse/KAFKA-12220
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Major
>
> We are migrating project from junit 4 to junit 5 (KAFKA-7339). PowerMock, 
> however, does not support junit 5 totally 
> (https://github.com/powermock/powermock/issues/830). Hence, we ought to 
> replace PowerMock by Mockito before migrating to junit 5 since rewriting all 
> tests which are depending on PowerMock can bring a bunch of changes.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-12220) Replace PowerMock by Mockito

[Ismael Juma (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-12220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-12220.
-
Resolution: Duplicate

The remaining connect tests that need to be migrated are being tracked via 
https://issues.apache.org/jira/browse/KAFKA-14132.

> Replace PowerMock by Mockito
> 
>
> Key: KAFKA-12220
> URL: https://issues.apache.org/jira/browse/KAFKA-12220
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Major
>
> We are migrating project from junit 4 to junit 5 (KAFKA-7339). PowerMock, 
> however, does not support junit 5 totally 
> (https://github.com/powermock/powermock/issues/830). Hence, we ought to 
> replace PowerMock by Mockito before migrating to junit 5 since rewriting all 
> tests which are depending on PowerMock can bring a bunch of changes.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

[Nick Telford]

Hi Lucas,

Yeah, this is pretty much the direction I'm thinking of going in now. You
make an interesting point about committing on-error under
ALOS/READ_COMMITTED, although I haven't had a chance to think through the
implications yet.

Something that I ran into earlier this week is an issue with the new
handling of TimeoutException. Without TX stores, TimeoutException under EOS
throws a TaskCorruptedException, which wipes the stores. However, with TX
stores, TimeoutException is now just bubbled up and dealt with as it is
under ALOS. The problem arises when the Producer#commitTransaction call
times out: Streams attempts to ignore the error and continue producing,
which causes the next call to Producer#send to throw
"IllegalStateException: Cannot attempt operation `send` because the
previous call to `commitTransaction` timed out and must be retried".

I'm not sure what we should do here: retrying the commitTransaction seems
logical, but what if it times out again? Where do we draw the line and
shutdown the instance?

Regards,
Nick

On Mon, 16 Oct 2023 at 13:19, Lucas Brutschy 
wrote:

> Hi all,
>
> I think I liked your suggestion of allowing EOS with READ_UNCOMMITTED,
> but keep wiping the state on error, and I'd vote for this solution
> when introducing `default.state.isolation.level`. This way, we'd have
> the most low-risk roll-out of this feature (no behavior change without
> reconfiguration), with the possibility of switching to the most sane /
> battle-tested default settings in 4.0. Essentially, we'd have a
> feature flag but call it `default.state.isolation.level` and don't
> have to deprecate it later.
>
> So the possible configurations would then be this:
>
> 1. ALOS/READ_UNCOMMITTED (default) = processing uses direct-to-DB, IQ
> reads from DB.
> 2. ALOS/READ_COMMITTED = processing uses WriteBatch, IQ reads from
> WriteBatch/DB. Flush on error (see note below).
> 3. EOS/READ_UNCOMMITTED (default) = processing uses direct-to-DB, IQ
> reads from DB. Wipe state on error.
> 4. EOS/READ_COMMITTED = processing uses WriteBatch, IQ reads from
> WriteBatch/DB.
>
> I believe the feature is important enough that we will see good
> adoption even without changing the default. In 4.0, when we have seen
> this being adopted and is battle-tested, we make READ_COMMITTED the
> default for EOS, or even READ_COMITTED always the default, depending
> on our experiences. And we could add a clever implementation of
> READ_UNCOMITTED with WriteBatches later.
>
> The only smell here is that `default.state.isolation.level` wouldn't
> be purely an IQ setting, but it would also (slightly) change the
> behavior of the processing, but that seems unavoidable as long as we
> haven't solve READ_UNCOMITTED IQ with WriteBatches.
>
> Minor: As for Bruno's point 4, I think if we are concerned about this
> behavior (we don't necessarily have to be, because it doesn't violate
> ALOS guarantees as far as I can see), we could make
> ALOS/READ_COMMITTED more similar to ALOS/READ_UNCOMITTED by flushing
> the WriteBatch on error (obviously, only if we have a chance to do
> that).
>
> Cheers,
> Lucas
>
> On Mon, Oct 16, 2023 at 12:19 PM Nick Telford 
> wrote:
> >
> > Hi Guozhang,
> >
> > The KIP as it stands introduces a new configuration,
> > default.state.isolation.level, which is independent of processing.mode.
> > It's intended that this new configuration be used to configure a global
> IQ
> > isolation level in the short term, with a future KIP introducing the
> > capability to change the isolation level on a per-query basis, falling
> back
> > to the "default" defined by this config. That's why I called it
> "default",
> > for future-proofing.
> >
> > However, it currently includes the caveat that READ_UNCOMMITTED is not
> > available under EOS. I think this is the coupling you are alluding to?
> >
> > This isn't intended to be a restriction of the API, but is currently a
> > technical limitation. However, after discussing with some users about
> > use-cases that would require READ_UNCOMMITTED under EOS, I'm inclined to
> > remove that clause and put in the necessary work to make that combination
> > possible now.
> >
> > I currently see two possible approaches:
> >
> >1. Disable TX StateStores internally when the IsolationLevel is
> >READ_UNCOMMITTED and the processing.mode is EOS. This is more
> difficult
> >than it sounds, as there are many assumptions being made throughout
> the
> >internals about the guarantees StateStores provide. It would
> definitely add
> >a lot of extra "if (read_uncommitted && eos)" branches, complicating
> >maintenance and testing.
> >2. Invest the time *now* to make READ_UNCOMMITTED of EOS StateStores
> >possible. I have some ideas on how this could be achieved, but they
> would
> >need testing and could introduce some additional issues. The benefit
> of
> >this approach is that it would make query-time IsolationLevels much
> simpler
> >to implement in the future.

[jira] [Resolved] (KAFKA-13414) Replace Powermock/EasyMock by Mockito in connect.storage package

[Ismael Juma (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-13414.
-
Resolution: Fixed

KafkaConfigBackingStoreTest is being tracked via KAFKA-14132, marking this as 
fixed.

> Replace Powermock/EasyMock by Mockito in connect.storage package
> 
>
> Key: KAFKA-13414
> URL: https://issues.apache.org/jira/browse/KAFKA-13414
> Project: Kafka
>  Issue Type: Sub-task
>  Components: KafkaConnect
>Reporter: Mickael Maison
>Assignee: Christo Lolov
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-989: RocksDB Iterator Metrics

[Nick Telford]

Hi Lucas,

Hmm, I'm not sure how we could reliably identify such leaked Iterators. If
we tried to include open iterators when calculating iterator-duration, we'd
need some kind of registry of all the open iterator creation timestamps,
wouldn't we?

In general, if you have a leaky Iterator, it should manifest as a
persistently climbing "open-iterators" metric, even on a busy node, because
each time that Iterator is used, it will leak another one. So even in the
presence of many non-leaky Iterators on a busy instance, the metric should
still consistently climb.

Regards,

Nick

On Mon, 16 Oct 2023 at 14:24, Lucas Brutschy 
wrote:

> Hi Nick!
>
> thanks for the KIP! I think this could be quite useful, given the
> problems that we had with leaks due to RocksDB resources not being
> closed.
>
> I don't have any pressing issues why we can't accept it like it is,
> just one minor point for discussion: would it also make sense to make
> it possible to identify a few very long-running / leaked iterators? I
> can imagine on a busy node, it would be hard to spot that 1% of the
> iterators never close when looking only at closed iterator or the
> number of iterators. But it could still be good to identify those
> leaks early. One option would be to add `iterator-duration-max` and
> take open iterators into account when computing the metric.
>
> Cheers,
> Lucas
>
>
> On Thu, Oct 5, 2023 at 3:50 PM Nick Telford 
> wrote:
> >
> > Hi Colt,
> >
> > I kept the details out of the KIP, because KIPs generally document
> > high-level design, but the way I'm doing it is like this:
> >
> > final ManagedKeyValueIterator
> > rocksDbPrefixSeekIterator = cf.prefixScan(accessor, prefixBytes);
> > --> final long startedAt = System.nanoTime();
> > openIterators.add(rocksDbPrefixSeekIterator);
> > rocksDbPrefixSeekIterator.onClose(() -> {
> > --> metricsRecorder.recordIteratorDuration(System.nanoTime() -
> > startedAt);
> > openIterators.remove(rocksDbPrefixSeekIterator);
> > });
> >
> > The lines with the arrow are the new code. This pattern is repeated
> > throughout RocksDBStore, wherever a new RocksDbIterator is created.
> >
> > Regards,
> > Nick
> >
> > On Thu, 5 Oct 2023 at 12:32, Colt McNealy  wrote:
> >
> > > Thank you for the KIP, Nick!
> > >
> > > This would be highly useful for many reasons. Much more sane than
> checking
> > > for leaked iterators by profiling memory usage while running 100's of
> > > thousands of range scans via interactive queries (:
> > >
> > > One small question:
> > >
> > > >The iterator-duration metrics will be updated whenever an Iterator's
> > > close() method is called
> > >
> > > Does the Iterator have its own "createdAt()" or equivalent field, or
> do we
> > > need to keep track of the Iterator's start time upon creation?
> > >
> > > Cheers,
> > > Colt McNealy
> > >
> > > *Founder, LittleHorse.dev*
> > >
> > >
> > > On Wed, Oct 4, 2023 at 9:07 AM Nick Telford 
> > > wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > KIP-989 is a small Kafka Streams KIP to add a few new metrics around
> the
> > > > creation and use of RocksDB Iterators, to aid users in identifying
> > > > "Iterator leaks" that could cause applications to leak native memory.
> > > >
> > > > Let me know what you think!
> > > >
> > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-989%3A+RocksDB+Iterator+Metrics
> > > >
> > > > P.S. I'm not too sure about the formatting of the "New Metrics"
> table,
> > > any
> > > > advice there would be appreciated.
> > > >
> > > > Regards,
> > > > Nick
> > > >
> > >
>

Re: [DISCUSS] KIP-988 Streams Standby Task Update Listener

[Eduwer Camacaro]

Hi Bruno,

Thanks for your observation; surely it will require a network call using
the admin client in order to know this "endOffset" and that will have an
impact on performance. We can either find a solution that has a low impact
on performance or ideally zero impact; unfortunately, I don't see a way to
have zero impact on performance. However, we can leverage the existing
#maybeUpdateLimitOffsetsForStandbyChangelogs method, which uses the admin
client to ask for these "endOffset"s. As far I can understand, this update
is done periodically using the "commit.interval.ms" configuration. I
believe this option will force us to invoke StandbyUpdateLister once this
interval is reached.

On Mon, Oct 16, 2023 at 8:52 AM Bruno Cadonna  wrote:

> Thanks for the KIP, Colt and Eduwer,
>
> Are you sure there is also not a significant performance impact for
> passing into the callback `currentEndOffset`?
>
> I am asking because the comment here:
>
> https://github.com/apache/kafka/blob/c32d2338a7e0079e539b74eb16f0095380a1ce85/streams/src/main/java/org/apache/kafka/streams/processor/internals/StoreChangelogReader.java#L129
>
> says that the end-offset is only updated once for standby tasks whose
> changelog topic is not piggy-backed on input topics. I could also not
> find the update of end-offset for those standbys.
>
>
> Best,
> Bruno
>
> On 10/16/23 10:55 AM, Lucas Brutschy wrote:
> > Hi all,
> >
> > it's a nice improvement! I don't have anything to add on top of the
> > previous comments, just came here to say that it seems to me consensus
> > has been reached and the result looks good to me.
> >
> > Thanks Colt and Eduwer!
> > Lucas
> >
> > On Sun, Oct 15, 2023 at 9:11 AM Colt McNealy 
> wrote:
> >>
> >> Thanks, Guozhang. I've updated the KIP and will start a vote.
> >>
> >> Colt McNealy
> >>
> >> *Founder, LittleHorse.dev*
> >>
> >>
> >> On Sat, Oct 14, 2023 at 10:27 AM Guozhang Wang <
> guozhang.wang...@gmail.com>
> >> wrote:
> >>
> >>> Thanks for the summary, that looks good to me.
> >>>
> >>> Guozhang
> >>>
> >>> On Fri, Oct 13, 2023 at 8:57 PM Colt McNealy 
> wrote:
> 
>  Hello there!
> 
>  Thanks everyone for the comments. There's a lot of back-and-forth
> going
> >>> on,
>  so I'll do my best to summarize what everyone's said in TLDR format:
> 
>  1. Rename `onStandbyUpdateStart()` -> `onUpdateStart()`,  and do
> >>> similarly
>  for the other methods.
>  2. Keep `SuspendReason.PROMOTED` and `SuspendReason.MIGRATED`.
>  3. Remove the `earliestOffset` parameter for performance reasons.
> 
>  If that's all fine with everyone, I'll update the KIP and we—well,
> mostly
>  Edu (:  —will open a PR.
> 
>  Cheers,
>  Colt McNealy
> 
>  *Founder, LittleHorse.dev*
> 
> 
>  On Fri, Oct 13, 2023 at 7:58 PM Eduwer Camacaro <
> edu...@littlehorse.io>
>  wrote:
> 
> > Hello everyone,
> >
> > Thanks for all your feedback for this KIP!
> >
> > I think that the key to choosing proper names for this API is
> >>> understanding
> > the terms used inside the StoreChangelogReader. Currently, this class
> >>> has
> > two possible states: ACTIVE_RESTORING and STANDBY_UPDATING. In my
> >>> opinion,
> > using StandbyUpdateListener for the interface fits better on these
> >>> terms.
> > Same applies for onUpdateStart/Suspended.
> >
> > StoreChangelogReader uses "the same mechanism" for active task
> >>> restoration
> > and standby task updates, but this is an implementation detail. Under
> > normal circumstances (no rebalances or task migrations), the
> changelog
> > reader will be in STANDBY_UPDATING, which means it will be updating
> >>> standby
> > tasks as long as there are new records in the changelog topic. That's
> >>> why I
> > prefer onStandbyUpdated instead of onBatchUpdated, even if it doesn't
> >>> 100%
> > align with StateRestoreListener, but either one is fine.
> >
> > Edu
> >
> > On Fri, Oct 13, 2023 at 8:53 PM Guozhang Wang <
> >>> guozhang.wang...@gmail.com>
> > wrote:
> >
> >> Hello Colt,
> >>
> >> Thanks for writing the KIP! I have read through the updated KIP and
> >> overall it looks great. I only have minor naming comments (well,
> >> aren't naming the least boring stuff to discuss and that takes the
> >> most of the time for KIPs :P):
> >>
> >> 1. I tend to agree with Sophie regarding whether or not to include
> >> "Standby" in the functions of "onStandbyUpdateStart/Suspended",
> since
> >> it is also more consistent with the functions of
> >> "StateRestoreListener" where we do not name it as
> >> "onStateRestoreState" etc.
> >>
> >> 2. I know in community discussions we sometimes say "a standby is
> >> promoted to active", but in the official code / java docs we did not
> >> have a term of "promotion", since what the code does is really
> >>> recycle
> >> the 

[jira] [Resolved] (KAFKA-7686) Remove PowerMock from Connect Tests

[Ismael Juma (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-7686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-7686.

Resolution: Duplicate

> Remove PowerMock from Connect Tests
> ---
>
> Key: KAFKA-7686
> URL: https://issues.apache.org/jira/browse/KAFKA-7686
> Project: Kafka
>  Issue Type: Task
>  Components: KafkaConnect
>Reporter: Magesh kumar Nandakumar
>Assignee: Magesh kumar Nandakumar
>Priority: Minor
>
> Remove PowerMock from Connect Tests



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Apache Kafka 3.5.2 release

[Bill Bejeck]

Thanks for driving the release, Luke.

+1
-Bill

On Tue, Oct 17, 2023 at 5:05 AM Satish Duggana 
wrote:

> Thanks Luke for volunteering for 3.5.2 release.
>
> On Tue, 17 Oct 2023 at 11:58, Josep Prat 
> wrote:
> >
> > Hi Luke,
> >
> > Thanks for taking this one!
> >
> > Best,
> >
> > On Tue, Oct 17, 2023 at 8:12 AM Luke Chen  wrote:
> >
> > > Hi all,
> > >
> > > I'd like to volunteer as release manager for the Apache Kafka 3.5.2, to
> > > have an important bug/vulnerability fix release for 3.5.1.
> > >
> > > If there are no objections, I'll start building a release plan in
> thewiki
> > > in the next couple of weeks.
> > >
> > > Thanks,
> > > Luke
> > >
> >
> >
> > --
> > [image: Aiven] 
> >
> > *Josep Prat*
> > Open Source Engineering Director, *Aiven*
> > josep.p...@aiven.io   |   +491715557497
> > aiven.io    |   <
> https://www.facebook.com/aivencloud>
> >      <
> https://twitter.com/aiven_io>
> > *Aiven Deutschland GmbH*
> > Alexanderufer 3-7, 10117 Berlin
> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > Amtsgericht Charlottenburg, HRB 209739 B
>

Re: [DISCUSS] KIP-971 Expose replication-offset-lag MirrorMaker2 metric

[Viktor Somogyi-Vass]

Hi hudeqi,

Good thinking about the OOM and resource leaks.
The "update.replication.lag.interval.time" I think is almost good but we
should include that it is about a metric (like
"replication.lag.interval.metric.update.time") so it's obvious without the
docs too.

Thanks,
Viktor

On Sat, Oct 7, 2023 at 8:53 AM hudeqi <16120...@bjtu.edu.cn> wrote:

> Hi, Elkhan, Viktor.
>
> I took a look at the updated KIP. I think Viktor mentioned that he did not
> see the relevant configuration, which refers to "(Optional) -
> MirrorConnectorConfig - a configuration to control the poll interval for
> the Consumer.endOffsets() call at LEO acquisition mentioned below". I think
> we can introduce the name of this configuration here, such as
> "update.replication.lag.interval.time", which means that in a separate
> periodic scheduling thread, the lag is calculated by this interval time
> through "consumer.endOffsets - LRO". In addition, for the LRO cache, you
> can add an expired time attribute for each partition. If this expired
> interval time is exceeded before next updated, the LRO of this partition
> can be removed from the cache to avoid possible leaks and OOM.
>
> best,
> hudeqi

Re: [DISCUSS] KIP-978: Allow dynamic reloading of certificates with different DN / SANs

[Viktor Somogyi-Vass]

Hi Jakub,

I think the KIP looks good overall, and I have one question for now.
Would it make sense to split the config you want to introduce
(ssl.allow.dn.and.san.changes) into two configs? Would users want to enable
one but not the other?

Thanks,
Viktor

On Wed, Sep 13, 2023 at 10:00 PM Jakub Scholz  wrote:

> Hi all,
>
> I would like to start the discussion about the KIP-978: Allow dynamic
> reloading of certificates with different DN / SANs
> <
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263429128
> >.
> It proposes adding an option to disable the current validation of the DN
> and SANs when dynamically changing the keystore. Please have a look and let
> me know your thoughts ...
>
> Thanks & Regards
> Jakub
>

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2296

[Apache Jenkins Server]

See 

Re: Re: [DISCUSS] KIP-986: Cross-Cluster Replication

[Viktor Somogyi-Vass]

Hi Greg,

> I don't think I understand what you mean here. Are you suggesting an
alternative to the Admin API? An external project could certainly
build such a component with the Admin API.

So I was thinking of something more complex than the Admin API, an external
service that can instruct clusters (using the Admin API) to start or stop
replication for certain partitions, list their replication flows, throttle
them and so on. With this you won't be able to control just one cluster but
you could register your clusters and control them in a centralized fashion.
Similar to what RestServer is to Connect, though with this you don't create
connectors but replication flows.

I've updated the KIP with my remote log storage thoughts (in data
semantics).

Best,
Viktor

On Sat, Oct 7, 2023 at 7:22 PM Greg Harris 
wrote:

> Hello hudeqi,
>
> I apologize if the KIP and discussion have diverged, as I've been
> trying to add detail rather than propose changes.
>
> > Why can't we use the follower fetch protocol?
>
> What you've described sounds like a very reasonable implementation of
> CCR. I purposely have not specified any implementation details so far
> and have been focusing only on the user-facing semantics of the
> feature. You are of course welcome to add details of how the
> follower-fetch implementation would work to the KIP.
>
> I think maybe this wording in the KIP was ambiguous: "Are not eligible
> for fetch-from-follower on the source cluster". I clarified the
> justification for this earlier in Tom's point D3. But to make the
> statement itself more clear:
>
> Consumers on the source cluster will not see a cross-cluster leader or
> followers as valid replicas to fetch from for load-sharing or latency
> optimization. Consumers on the target cluster will see the
> cross-cluster followers as valid replicas to fetch from, as they will
> appear as normal replicas. This was only meant to describe the
> relationship of the remote replicas with Consumers, not with each
> other.
>
> I hope this is more clear.
> Greg
>
> On Sat, Oct 7, 2023 at 1:38 AM hudeqi <16120...@bjtu.edu.cn> wrote:
> >
> > Hi, Greg.
> >
> > After reading this KIP and your discussion, I feel that it is very
> divergent. I can only start from one of them:
> > Why can't we use the follower fetch protocol? The leader of the target
> cluster topic partition can be treated as the follower of the source
> cluster topic partition leader, and the fetched data is directly appended
> to the local log (the remote fetch thread is inherited to the follower
> fetch thread, thereby retaining the offset of the log), so that consumer/
> producer client can be omitted. Of course, this is just data replication. I
> may have to think more about group offset/acl/config replication.
> >
> > best,
> > hudeqi
>

[jira] [Created] (KAFKA-15622) Delete configs deprecated by KIP-629

[Mickael Maison (Jira)]

Mickael Maison created KAFKA-15622:
--

 Summary: Delete configs deprecated by KIP-629
 Key: KAFKA-15622
 URL: https://issues.apache.org/jira/browse/KAFKA-15622
 Project: Kafka
  Issue Type: Improvement
Affects Versions: 4.0.0
Reporter: Mickael Maison
Assignee: Mickael Maison


[KIP-629|https://cwiki.apache.org/confluence/display/KAFKA/KIP-629%3A+Use+racially+neutral+terms+in+our+codebase]
 deprecated a bunch of configurations. We should delete them in the next major 
release.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Apache Kafka 3.5.2 release

[Satish Duggana]

Thanks Luke for volunteering for 3.5.2 release.

On Tue, 17 Oct 2023 at 11:58, Josep Prat  wrote:
>
> Hi Luke,
>
> Thanks for taking this one!
>
> Best,
>
> On Tue, Oct 17, 2023 at 8:12 AM Luke Chen  wrote:
>
> > Hi all,
> >
> > I'd like to volunteer as release manager for the Apache Kafka 3.5.2, to
> > have an important bug/vulnerability fix release for 3.5.1.
> >
> > If there are no objections, I'll start building a release plan in thewiki
> > in the next couple of weeks.
> >
> > Thanks,
> > Luke
> >
>
>
> --
> [image: Aiven] 
>
> *Josep Prat*
> Open Source Engineering Director, *Aiven*
> josep.p...@aiven.io   |   +491715557497
> aiven.io    |   
>      
> *Aiven Deutschland GmbH*
> Alexanderufer 3-7, 10117 Berlin
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> Amtsgericht Charlottenburg, HRB 209739 B

[jira] [Created] (KAFKA-15621) Add histogram metrics to GroupCoordinatorRuntimeMetrics

[David Jacot (Jira)]

David Jacot created KAFKA-15621:
---

 Summary: Add histogram metrics to GroupCoordinatorRuntimeMetrics
 Key: KAFKA-15621
 URL: https://issues.apache.org/jira/browse/KAFKA-15621
 Project: Kafka
  Issue Type: Sub-task
Reporter: David Jacot


We will add new histograms to Kafka Metrics library soon. When we have it, we 
can start using it in GroupCoordinatorRuntimeMetrics.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-985 Add reverseRange and reverseAll query over kv-store in IQv2

[Lucas Brutschy]

+1 (binding)

Thanks for the KIP!

On Tue, Oct 17, 2023 at 2:31 AM Matthias J. Sax  wrote:
>
> +1 (binding)
>
>
> On 10/13/23 9:24 AM, Hanyu (Peter) Zheng wrote:
> > Hello everyone,
> >
> > I would like to start a vote for KIP-985 that Add reverseRange and
> > reverseAll query over kv-store in IQv2.
> >
> > Sincerely,
> > Hanyu
> >
> > On Fri, Oct 13, 2023 at 9:15 AM Hanyu (Peter) Zheng 
> > wrote:
> >
> >>
> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-985:+Add+reverseRange+and+reverseAll+query+over+kv-store+in+IQv2
> >>
> >> --
> >>
> >> [image: Confluent] 
> >> Hanyu (Peter) Zheng he/him/his
> >> Software Engineer Intern
> >> +1 (213) 431-7193 <+1+(213)+431-7193>
> >> Follow us: [image: Blog]
> >> [image:
> >> Twitter] [image: LinkedIn]
> >> [image: Slack]
> >> [image: YouTube]
> >> 
> >>
> >> [image: Try Confluent Cloud for Free]
> >> 
> >>
> >
> >

[jira] [Created] (KAFKA-15620) Duplicate remote log DELETE_SEGMENT metadata is generated when there are multiple leader epochs in the segment

[Henry Cai (Jira)]

Henry Cai created KAFKA-15620:
-

 Summary: Duplicate remote log DELETE_SEGMENT metadata is generated 
when there are multiple leader epochs in the segment
 Key: KAFKA-15620
 URL: https://issues.apache.org/jira/browse/KAFKA-15620
 Project: Kafka
  Issue Type: Bug
  Components: log
Affects Versions: 3.6.0
Reporter: Henry Cai
 Fix For: 3.6.1


Use the newly released 3.6.0, turn on tiered storage feature: 

{*}remote.log.storage.system.enable{*}=true

Set up topic tier5 to be remote storage enabled.  Adding some data to the topic 
and the data is replicated to remote storage.  After a few days when the log 
segment is removed from remote storage when log retention kicks in, noticed the 
following warnings in the server log:

[2023-10-16 22:19:10,971] DEBUG Updating remote log segment metadata: 
[RemoteLogSegmentMetadataUpdate\{remoteLogSegmentId=RemoteLogSegmentId{topicIdPartition=QelVeVmER5CkjrzIiF07PQ:tier5-0,
 id=YFNCaWjPQFSKCngQ1QcKpA}, customMetadata=Optional.empty, 
state=DELETE_SEGMENT_STARTED, eventTimestampMs=1697005926358, brokerId=1043}] 
(org.apache.kafka.server.log.remote.metadata.storage.RemoteLogMetadataCache)

[2023-10-16 22:19:10,971] WARN Error occurred while updating the remote log 
segment. 
(org.apache.kafka.server.log.remote.metadata.storage.RemotePartitionMetadataStore)

org.apache.kafka.server.log.remote.storage.RemoteResourceNotFoundException: No 
remote log segment metadata found for 
:RemoteLogSegmentId\{topicIdPartition=QelVeVmER5CkjrzIiF07PQ:tier5-0, 
id=YFNCaWjPQFSKCngQ1QcKpA}

        at 
org.apache.kafka.server.log.remote.metadata.storage.RemoteLogMetadataCache.updateRemoteLogSegmentMetadata(RemoteLogMetadataCache.java:161)

        at 
org.apache.kafka.server.log.remote.metadata.storage.RemotePartitionMetadataStore.handleRemoteLogSegmentMetadataUpdate(RemotePartitionMetadataStore.java:89)

        at 
org.apache.kafka.server.log.remote.metadata.storage.RemotePartitionMetadataEventHandler.handleRemoteLogMetadata(RemotePartitionMetadataEventHandler.java:33)

        at 
org.apache.kafka.server.log.remote.metadata.storage.ConsumerTask.processConsumerRecord(ConsumerTask.java:157)

        at 
org.apache.kafka.server.log.remote.metadata.storage.ConsumerTask.run(ConsumerTask.java:133)

        at java.base/java.lang.Thread.run(Thread.java:829)

 

After some debugging, realized the problem is there are 2 sets of 
DELETE_SEGMENT_STARTED/FINISHED pairs in the remote metadata topic for this 
segment.  And traced the log to where the log retention started and saw there 
are two delete log segment happened at that time:

 

[2023-10-10 23:32:05,929] INFO [RemoteLogManager=1043 
partition=QelVeVmER5CkjrzIiF07PQ:tier5-0] About to delete remote log segment 
RemoteLogSegmentId{topicIdPartition=QelVeVmER5CkjrzIiF07PQ:tier5-0, 
id={*}YFNCaWjPQFSKCngQ1QcKpA{*}} due to retention time 60480ms breach based 
on the largest record timestamp in the segment 
(kafka.log.remote.RemoteLogManager$RLMTask)

[2023-10-10 23:32:05,929] INFO [RemoteLogManager=1043 
partition=QelVeVmER5CkjrzIiF07PQ:tier5-0] About to delete remote log segment 
RemoteLogSegmentId{topicIdPartition=QelVeVmER5CkjrzIiF07PQ:tier5-0, 
id={*}YFNCaWjPQFSKCngQ1QcKpA{*}} due to retention time 60480ms breach based 
on the largest record timestamp in the segment 
(kafka.log.remote.RemoteLogManager$RLMTask)

 

And dumped out the content of the original COPY_SEGMENT_STARTED for this 
segment (which triggers the generation of the later DELETE_SEGMENT metadata):

[2023-10-16 22:19:10,894] DEBUG Adding to in-progress state: 
[RemoteLogSegmentMetadata\{remoteLogSegmentId=RemoteLogSegmentId{topicIdPartition=QelVeVmER5CkjrzIiF07PQ:tier5-0,
 id=YFNCaWjPQFSKCngQ1QcKpA}, startOffset=6387830, endOffset=9578660, 
brokerId=1043, maxTimestampMs=1696401123036, eventTimestampMs=1696401127062, 
segmentLeaderEpochs=\{5=6387830, 6=6721329}, segmentSizeInBytes=511987531, 
customMetadata=Optional.empty, state=COPY_SEGMENT_STARTED}] 
(org.apache.kafka.server.log.remote.metadata.storage.RemoteLogMetadataCache)

 

You can see there are 2 leader epochs in this segment: 
segmentLeaderEpochs=\{5=6387830, 6=6721329}

>From the remote log retention code 
>([https://github.com/apache/kafka/blob/3.6.0/core/src/main/java/kafka/log/remote/RemoteLogManager.java#L987)]

It's bucketing segments into epochs first and then looping through epochs.

I am not sure whether it should generate one or two DELETE_SEGMENT for this 
COPY_SEGMENT_START segment.  If it needs to generate 2 DELETE_SEGMENT metadata, 
the consumer task needs to handle that duplicate metadata situation (not 
throwing exceptions in the log).

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #2295

[Apache Jenkins Server]

See 

Re: [DISCUSS] Apache Kafka 3.5.2 release

[Josep Prat]

Hi Luke,

Thanks for taking this one!

Best,

On Tue, Oct 17, 2023 at 8:12 AM Luke Chen  wrote:

> Hi all,
>
> I'd like to volunteer as release manager for the Apache Kafka 3.5.2, to
> have an important bug/vulnerability fix release for 3.5.1.
>
> If there are no objections, I'll start building a release plan in thewiki
> in the next couple of weeks.
>
> Thanks,
> Luke
>


-- 
[image: Aiven] 

*Josep Prat*
Open Source Engineering Director, *Aiven*
josep.p...@aiven.io   |   +491715557497
aiven.io    |   
     
*Aiven Deutschland GmbH*
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
Amtsgericht Charlottenburg, HRB 209739 B

[DISCUSS] Apache Kafka 3.5.2 release

[Luke Chen]

Hi all,

I'd like to volunteer as release manager for the Apache Kafka 3.5.2, to
have an important bug/vulnerability fix release for 3.5.1.

If there are no objections, I'll start building a release plan in thewiki
in the next couple of weeks.

Thanks,
Luke