Re: [ANNOUNCE] Apache Kafka 3.4.1

[Federico Valeri]

Thanks Luke!

On Wed, Jun 7, 2023 at 5:56 AM Kamal Chandraprakash
 wrote:
>
> Thanks Luke for running this release!
>
> On Wed, Jun 7, 2023 at 8:08 AM Chia-Ping Tsai  wrote:
>
> > Thank Luke for this hard work!!!
> >
> > > Chris Egerton  於 2023年6月7日 上午10:35 寫道：
> > >
> > > Thanks for running this release, Luke!
> > >
> > > On Tue, Jun 6, 2023, 22:31 Luke Chen  wrote:
> > >
> > >> The Apache Kafka community is pleased to announce the release for
> > >> Apache Kafka 3.4.1.
> > >>
> > >> This is a bug fix release and it includes fixes and improvements from
> > >> 58 JIRAs, including a few critical bugs:
> > >> - core
> > >> KAFKA-14644 Process should stop after failure in raft IO thread
> > >> KAFKA-14946 KRaft controller node shutting down while renouncing
> > leadership
> > >> KAFKA-14887 ZK session timeout can cause broker to shutdown
> > >> - client
> > >> KAFKA-14639 Kafka CooperativeStickyAssignor revokes/assigns partition
> > >> in one rebalance cycle
> > >> - connect
> > >> KAFKA-12558 MM2 may not sync partition offsets correctly
> > >> KAFKA-14666 MM2 should translate consumer group offsets behind
> > replication
> > >> flow
> > >> - stream
> > >> KAFKA-14172 bug: State stores lose state when tasks are reassigned under
> > >> EOS
> > >>
> > >> All of the changes in this release can be found in the release notes:
> > >>
> > >> https://www.apache.org/dist/kafka/3.4.1/RELEASE_NOTES.html
> > >>
> > >> You can download the source and binary release (Scala 2.12 and Scala
> > 2.13)
> > >> from:
> > >>
> > >> https://kafka.apache.org/downloads#3.4.1
> > >>
> > >>
> > >>
> > ---
> > >>
> > >> Apache Kafka is a distributed streaming platform with four core APIs:
> > >>
> > >> ** The Producer API allows an application to publish a stream records
> > >> to one or more Kafka topics.
> > >>
> > >> ** The Consumer API allows an application to subscribe to one or more
> > >> topics and process the stream of records produced to them.
> > >>
> > >> ** The Streams API allows an application to act as a stream processor,
> > >> consuming an input stream from one or more topics and producing an
> > >> output stream to one or more output topics, effectively transforming
> > >> the input streams to output streams.
> > >>
> > >> ** The Connector API allows building and running reusable producers or
> > >> consumers that connect Kafka topics to existing applications or data
> > >> systems. For example, a connector to a relational database might
> > >> capture every change to a table.
> > >>
> > >>
> > >> With these APIs, Kafka can be used for two broad classes of application:
> > >>
> > >> ** Building real-time streaming data pipelines that reliably get data
> > >> between systems or applications.
> > >>
> > >> ** Building real-time streaming applications that transform or react
> > >> to the streams of data.
> > >>
> > >> Apache Kafka is in use at large and small companies worldwide,
> > >> including Capital One, Goldman Sachs, ING, LinkedIn, Netflix,
> > >> Pinterest, Rabobank, Target, The New York Times, Uber, Yelp, and
> > >> Zalando, among others.
> > >>
> > >> A big thank you for the following 32 contributors to this release!
> > >>
> > >> atu-sharm, Chia-Ping Tsai, Chris Egerton, Colin Patrick McCabe,
> > >> csolidum, David Arthur, David Jacot, Divij Vaidya, egyedt,
> > >> emilnkrastev, Eric Haag, Greg Harris, Guozhang Wang, Hector Geraldino,
> > >> hudeqi, Jason Gustafson, Jeff Kim, Jorge Esteban Quilcate Otoya, José
> > >> Armando García Sancio, Lucia Cerchie, Luke Chen, Manikumar Reddy,
> > >> Matthias J. Sax, Mickael Maison, Philip Nee, Purshotam Chauhan, Rajini
> > >> Sivaram, Ron Dagostino, Terry, Victoria Xia, Viktor Somogyi-Vass, Yash
> > >> Mayya
> > >>
> > >> We welcome your help and feedback. For more information on how to
> > >> report problems, and to get involved, visit the project website at
> > >> https://kafka.apache.org/
> > >>
> > >>
> > >> Thank you!
> > >>
> > >> Regards,
> > >> Luke
> > >>
> >
> >

Re: [ANNOUNCE] Apache Kafka 3.4.1

[Josep Prat]

Thanks for running the release!

Best,

———
Josep Prat

Aiven Deutschland GmbH

Alexanderufer 3-7, 10117 Berlin

Amtsgericht Charlottenburg, HRB 209739 B

Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen

m: +491715557497

w: aiven.io

e: josep.p...@aiven.io

On Wed, Jun 7, 2023, 05:57 Kamal Chandraprakash <
kamal.chandraprak...@gmail.com> wrote:

> Thanks Luke for running this release!
>
> On Wed, Jun 7, 2023 at 8:08 AM Chia-Ping Tsai  wrote:
>
> > Thank Luke for this hard work!!!
> >
> > > Chris Egerton  於 2023年6月7日 上午10:35 寫道：
> > >
> > > Thanks for running this release, Luke!
> > >
> > > On Tue, Jun 6, 2023, 22:31 Luke Chen  wrote:
> > >
> > >> The Apache Kafka community is pleased to announce the release for
> > >> Apache Kafka 3.4.1.
> > >>
> > >> This is a bug fix release and it includes fixes and improvements from
> > >> 58 JIRAs, including a few critical bugs:
> > >> - core
> > >> KAFKA-14644 Process should stop after failure in raft IO thread
> > >> KAFKA-14946 KRaft controller node shutting down while renouncing
> > leadership
> > >> KAFKA-14887 ZK session timeout can cause broker to shutdown
> > >> - client
> > >> KAFKA-14639 Kafka CooperativeStickyAssignor revokes/assigns partition
> > >> in one rebalance cycle
> > >> - connect
> > >> KAFKA-12558 MM2 may not sync partition offsets correctly
> > >> KAFKA-14666 MM2 should translate consumer group offsets behind
> > replication
> > >> flow
> > >> - stream
> > >> KAFKA-14172 bug: State stores lose state when tasks are reassigned
> under
> > >> EOS
> > >>
> > >> All of the changes in this release can be found in the release notes:
> > >>
> > >> https://www.apache.org/dist/kafka/3.4.1/RELEASE_NOTES.html
> > >>
> > >> You can download the source and binary release (Scala 2.12 and Scala
> > 2.13)
> > >> from:
> > >>
> > >> https://kafka.apache.org/downloads#3.4.1
> > >>
> > >>
> > >>
> >
> ---
> > >>
> > >> Apache Kafka is a distributed streaming platform with four core APIs:
> > >>
> > >> ** The Producer API allows an application to publish a stream records
> > >> to one or more Kafka topics.
> > >>
> > >> ** The Consumer API allows an application to subscribe to one or more
> > >> topics and process the stream of records produced to them.
> > >>
> > >> ** The Streams API allows an application to act as a stream processor,
> > >> consuming an input stream from one or more topics and producing an
> > >> output stream to one or more output topics, effectively transforming
> > >> the input streams to output streams.
> > >>
> > >> ** The Connector API allows building and running reusable producers or
> > >> consumers that connect Kafka topics to existing applications or data
> > >> systems. For example, a connector to a relational database might
> > >> capture every change to a table.
> > >>
> > >>
> > >> With these APIs, Kafka can be used for two broad classes of
> application:
> > >>
> > >> ** Building real-time streaming data pipelines that reliably get data
> > >> between systems or applications.
> > >>
> > >> ** Building real-time streaming applications that transform or react
> > >> to the streams of data.
> > >>
> > >> Apache Kafka is in use at large and small companies worldwide,
> > >> including Capital One, Goldman Sachs, ING, LinkedIn, Netflix,
> > >> Pinterest, Rabobank, Target, The New York Times, Uber, Yelp, and
> > >> Zalando, among others.
> > >>
> > >> A big thank you for the following 32 contributors to this release!
> > >>
> > >> atu-sharm, Chia-Ping Tsai, Chris Egerton, Colin Patrick McCabe,
> > >> csolidum, David Arthur, David Jacot, Divij Vaidya, egyedt,
> > >> emilnkrastev, Eric Haag, Greg Harris, Guozhang Wang, Hector Geraldino,
> > >> hudeqi, Jason Gustafson, Jeff Kim, Jorge Esteban Quilcate Otoya, José
> > >> Armando García Sancio, Lucia Cerchie, Luke Chen, Manikumar Reddy,
> > >> Matthias J. Sax, Mickael Maison, Philip Nee, Purshotam Chauhan, Rajini
> > >> Sivaram, Ron Dagostino, Terry, Victoria Xia, Viktor Somogyi-Vass, Yash
> > >> Mayya
> > >>
> > >> We welcome your help and feedback. For more information on how to
> > >> report problems, and to get involved, visit the project website at
> > >> https://kafka.apache.org/
> > >>
> > >>
> > >> Thank you!
> > >>
> > >> Regards,
> > >> Luke
> > >>
> >
> >
>

Re: [DISCUSS] KIP-935: Extend AlterConfigPolicy with existing configurations

[Colin McCabe]

On Tue, Jun 6, 2023, at 06:57, Jorge Esteban Quilcate Otoya wrote:
> Thanks Colin.
>
>> I would suggest renaming the "configs" parameter to "proposedConfigs," in
> both the new and old RequestMetadata constructors, to make things clearer.
> This would be a binary and API-compatible change in Java.
>
> Sure, fully agree. KIP is updated with this suggestion.
>
>> We should also clarify that if configurations are being proposed for
> deletion, they won't appear in proposedConfigs.
>
> Great catch. Wasn't aware of this, but I think it's valuable for the API to
> also include the list of configurations to be deleted.
> For this, I have extended the `RequestMetadata` type with a list of
> proposed configs to delete:
>

Hi Jorge,

Thanks for the reply.

Rather than having a separate list of proposedConfigsToDelete, it seems like we 
could have an accessor function that calculates this when needed. After all, 
it's completely determined by existingConfigs and proposedConfigs. And some 
plugins will want the list and some won't (or will want to do a slightly 
different analysis)

regards,
Colin


> ```
> class RequestMetadata {
>
> private final ConfigResource resource;
> private final Map proposedConfigs;
> private final List proposedConfigsToDelete;
> private final Map existingConfigs;
> ```
>
> Looking forward to your feedback.
>
> Cheers,
> Jorge.
>
> On Fri, 2 Jun 2023 at 22:42, Colin McCabe  wrote:
>
>> Hi Jorge,
>>
>> This is a good KIP which addresses a real gap we have today.
>>
>> I would suggest renaming the "configs" parameter to "proposedConfigs," in
>> both the new and old RequestMetadata constructors, to make things clearer.
>> This would be a binary and API-compatible change in Java. We should also
>> clarify that if configurations are being proposed for deletion, they won't
>> appear in proposedConfigs.
>>
>> best,
>> Colin
>>
>>
>> On Tue, May 23, 2023, at 03:03, Christo Lolov wrote:
>> > Hello!
>> >
>> > This proposal will address problems with configuration dependencies
>> which I
>> > run into very frequently, so I am fully supporting the development of
>> this
>> > feature!
>> >
>> > Best,
>> > Christo
>> >
>> > On Mon, 22 May 2023 at 17:18, Jorge Esteban Quilcate Otoya <
>> > quilcate.jo...@gmail.com> wrote:
>> >
>> >> Hi everyone,
>> >>
>> >> I'd like to start a discussion for KIP-935 <
>> >>
>> >>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-935%3A+Extend+AlterConfigPolicy+with+existing+configurations
>> >> >
>> >> which proposes extend AlterConfigPolicy with existing configuration to
>> >> enable more complex policies.
>> >>
>> >> There have been related KIPs in the past that haven't been accepted and
>> >> seem retired/abandoned as outlined in the motivation.
>> >> The scope of this KIP intends to be more specific to get most of the
>> >> benefits from previous discussions; and if previous KIPs are
>> resurrected,
>> >> should still be possible to do it if this one is adopted.
>> >>
>> >> Looking forward to your feedback!
>> >>
>> >> Thanks,
>> >> Jorge.
>> >>
>>

Re: [VOTE] KIP-938: Add more metrics for measuring KRaft performance

[Colin McCabe]

Hi all,

I added two new metrics to the list:

* LatestSnapshotGeneratedBytes
* LatestSnapshotGeneratedAgeMs

These will help monitor the period snapshot generation process.

best,
Colin


On Tue, Jun 6, 2023, at 22:21, Colin McCabe wrote:
> Hi Divij,
>
> Yes, I am referring to the feature level. I changed the description of 
> CurrentMetadataVersion to reference the feature level specifically.
>
> best,
> Colin
>
>
> On Tue, Jun 6, 2023, at 05:56, Divij Vaidya wrote:
>> "Each metadata version has a corresponding integer in the
>> MetadataVersion.java file."
>>
>> Please correct me if I'm wrong, but are you referring to "featureLevel" 
>> in
>> the enum at
>> https://github.com/apache/kafka/blob/trunk/server-common/src/main/java/org/apache/kafka/server/common/MetadataVersion.java#L45
>> ? Is yes, can we please update the description of the metric to make it
>> easier for the users to understand this? For example, we can say,
>> "Represents the current metadata version as an integer value. See
>> MetadataVersion (hyperlink) for a mapping between string and integer
>> formats of metadata version".
>>
>> --
>> Divij Vaidya
>>
>>
>>
>> On Tue, Jun 6, 2023 at 1:51 PM Ron Dagostino  wrote:
>>
>>> Thanks again for the KIP, Colin.  +1 (binding).
>>>
>>> Ron
>>>
>>> > On Jun 6, 2023, at 7:02 AM, Igor Soarez 
>>> wrote:
>>> >
>>> > Thanks for the KIP.
>>> >
>>> > Seems straightforward, LGTM.
>>> > Non binding +1.
>>> >
>>> > --
>>> > Igor
>>> >
>>>

Re: [VOTE] KIP-938: Add more metrics for measuring KRaft performance

[Colin McCabe]

Hi Divij,

Yes, I am referring to the feature level. I changed the description of 
CurrentMetadataVersion to reference the feature level specifically.

best,
Colin


On Tue, Jun 6, 2023, at 05:56, Divij Vaidya wrote:
> "Each metadata version has a corresponding integer in the
> MetadataVersion.java file."
>
> Please correct me if I'm wrong, but are you referring to "featureLevel" 
> in
> the enum at
> https://github.com/apache/kafka/blob/trunk/server-common/src/main/java/org/apache/kafka/server/common/MetadataVersion.java#L45
> ? Is yes, can we please update the description of the metric to make it
> easier for the users to understand this? For example, we can say,
> "Represents the current metadata version as an integer value. See
> MetadataVersion (hyperlink) for a mapping between string and integer
> formats of metadata version".
>
> --
> Divij Vaidya
>
>
>
> On Tue, Jun 6, 2023 at 1:51 PM Ron Dagostino  wrote:
>
>> Thanks again for the KIP, Colin.  +1 (binding).
>>
>> Ron
>>
>> > On Jun 6, 2023, at 7:02 AM, Igor Soarez 
>> wrote:
>> >
>> > Thanks for the KIP.
>> >
>> > Seems straightforward, LGTM.
>> > Non binding +1.
>> >
>> > --
>> > Igor
>> >
>>

Re: [ANNOUNCE] Apache Kafka 3.4.1

[Kamal Chandraprakash]

Thanks Luke for running this release!

On Wed, Jun 7, 2023 at 8:08 AM Chia-Ping Tsai  wrote:

> Thank Luke for this hard work!!!
>
> > Chris Egerton  於 2023年6月7日 上午10:35 寫道：
> >
> > Thanks for running this release, Luke!
> >
> > On Tue, Jun 6, 2023, 22:31 Luke Chen  wrote:
> >
> >> The Apache Kafka community is pleased to announce the release for
> >> Apache Kafka 3.4.1.
> >>
> >> This is a bug fix release and it includes fixes and improvements from
> >> 58 JIRAs, including a few critical bugs:
> >> - core
> >> KAFKA-14644 Process should stop after failure in raft IO thread
> >> KAFKA-14946 KRaft controller node shutting down while renouncing
> leadership
> >> KAFKA-14887 ZK session timeout can cause broker to shutdown
> >> - client
> >> KAFKA-14639 Kafka CooperativeStickyAssignor revokes/assigns partition
> >> in one rebalance cycle
> >> - connect
> >> KAFKA-12558 MM2 may not sync partition offsets correctly
> >> KAFKA-14666 MM2 should translate consumer group offsets behind
> replication
> >> flow
> >> - stream
> >> KAFKA-14172 bug: State stores lose state when tasks are reassigned under
> >> EOS
> >>
> >> All of the changes in this release can be found in the release notes:
> >>
> >> https://www.apache.org/dist/kafka/3.4.1/RELEASE_NOTES.html
> >>
> >> You can download the source and binary release (Scala 2.12 and Scala
> 2.13)
> >> from:
> >>
> >> https://kafka.apache.org/downloads#3.4.1
> >>
> >>
> >>
> ---
> >>
> >> Apache Kafka is a distributed streaming platform with four core APIs:
> >>
> >> ** The Producer API allows an application to publish a stream records
> >> to one or more Kafka topics.
> >>
> >> ** The Consumer API allows an application to subscribe to one or more
> >> topics and process the stream of records produced to them.
> >>
> >> ** The Streams API allows an application to act as a stream processor,
> >> consuming an input stream from one or more topics and producing an
> >> output stream to one or more output topics, effectively transforming
> >> the input streams to output streams.
> >>
> >> ** The Connector API allows building and running reusable producers or
> >> consumers that connect Kafka topics to existing applications or data
> >> systems. For example, a connector to a relational database might
> >> capture every change to a table.
> >>
> >>
> >> With these APIs, Kafka can be used for two broad classes of application:
> >>
> >> ** Building real-time streaming data pipelines that reliably get data
> >> between systems or applications.
> >>
> >> ** Building real-time streaming applications that transform or react
> >> to the streams of data.
> >>
> >> Apache Kafka is in use at large and small companies worldwide,
> >> including Capital One, Goldman Sachs, ING, LinkedIn, Netflix,
> >> Pinterest, Rabobank, Target, The New York Times, Uber, Yelp, and
> >> Zalando, among others.
> >>
> >> A big thank you for the following 32 contributors to this release!
> >>
> >> atu-sharm, Chia-Ping Tsai, Chris Egerton, Colin Patrick McCabe,
> >> csolidum, David Arthur, David Jacot, Divij Vaidya, egyedt,
> >> emilnkrastev, Eric Haag, Greg Harris, Guozhang Wang, Hector Geraldino,
> >> hudeqi, Jason Gustafson, Jeff Kim, Jorge Esteban Quilcate Otoya, José
> >> Armando García Sancio, Lucia Cerchie, Luke Chen, Manikumar Reddy,
> >> Matthias J. Sax, Mickael Maison, Philip Nee, Purshotam Chauhan, Rajini
> >> Sivaram, Ron Dagostino, Terry, Victoria Xia, Viktor Somogyi-Vass, Yash
> >> Mayya
> >>
> >> We welcome your help and feedback. For more information on how to
> >> report problems, and to get involved, visit the project website at
> >> https://kafka.apache.org/
> >>
> >>
> >> Thank you!
> >>
> >> Regards,
> >> Luke
> >>
>
>

Re: [ANNOUNCE] Apache Kafka 3.4.1

[Chia-Ping Tsai]

Thank Luke for this hard work!!!

> Chris Egerton  於 2023年6月7日 上午10:35 寫道：
> 
> Thanks for running this release, Luke!
> 
> On Tue, Jun 6, 2023, 22:31 Luke Chen  wrote:
> 
>> The Apache Kafka community is pleased to announce the release for
>> Apache Kafka 3.4.1.
>> 
>> This is a bug fix release and it includes fixes and improvements from
>> 58 JIRAs, including a few critical bugs:
>> - core
>> KAFKA-14644 Process should stop after failure in raft IO thread
>> KAFKA-14946 KRaft controller node shutting down while renouncing leadership
>> KAFKA-14887 ZK session timeout can cause broker to shutdown
>> - client
>> KAFKA-14639 Kafka CooperativeStickyAssignor revokes/assigns partition
>> in one rebalance cycle
>> - connect
>> KAFKA-12558 MM2 may not sync partition offsets correctly
>> KAFKA-14666 MM2 should translate consumer group offsets behind replication
>> flow
>> - stream
>> KAFKA-14172 bug: State stores lose state when tasks are reassigned under
>> EOS
>> 
>> All of the changes in this release can be found in the release notes:
>> 
>> https://www.apache.org/dist/kafka/3.4.1/RELEASE_NOTES.html
>> 
>> You can download the source and binary release (Scala 2.12 and Scala 2.13)
>> from:
>> 
>> https://kafka.apache.org/downloads#3.4.1
>> 
>> 
>> ---
>> 
>> Apache Kafka is a distributed streaming platform with four core APIs:
>> 
>> ** The Producer API allows an application to publish a stream records
>> to one or more Kafka topics.
>> 
>> ** The Consumer API allows an application to subscribe to one or more
>> topics and process the stream of records produced to them.
>> 
>> ** The Streams API allows an application to act as a stream processor,
>> consuming an input stream from one or more topics and producing an
>> output stream to one or more output topics, effectively transforming
>> the input streams to output streams.
>> 
>> ** The Connector API allows building and running reusable producers or
>> consumers that connect Kafka topics to existing applications or data
>> systems. For example, a connector to a relational database might
>> capture every change to a table.
>> 
>> 
>> With these APIs, Kafka can be used for two broad classes of application:
>> 
>> ** Building real-time streaming data pipelines that reliably get data
>> between systems or applications.
>> 
>> ** Building real-time streaming applications that transform or react
>> to the streams of data.
>> 
>> Apache Kafka is in use at large and small companies worldwide,
>> including Capital One, Goldman Sachs, ING, LinkedIn, Netflix,
>> Pinterest, Rabobank, Target, The New York Times, Uber, Yelp, and
>> Zalando, among others.
>> 
>> A big thank you for the following 32 contributors to this release!
>> 
>> atu-sharm, Chia-Ping Tsai, Chris Egerton, Colin Patrick McCabe,
>> csolidum, David Arthur, David Jacot, Divij Vaidya, egyedt,
>> emilnkrastev, Eric Haag, Greg Harris, Guozhang Wang, Hector Geraldino,
>> hudeqi, Jason Gustafson, Jeff Kim, Jorge Esteban Quilcate Otoya, José
>> Armando García Sancio, Lucia Cerchie, Luke Chen, Manikumar Reddy,
>> Matthias J. Sax, Mickael Maison, Philip Nee, Purshotam Chauhan, Rajini
>> Sivaram, Ron Dagostino, Terry, Victoria Xia, Viktor Somogyi-Vass, Yash
>> Mayya
>> 
>> We welcome your help and feedback. For more information on how to
>> report problems, and to get involved, visit the project website at
>> https://kafka.apache.org/
>> 
>> 
>> Thank you!
>> 
>> Regards,
>> Luke
>> 

Re: [ANNOUNCE] Apache Kafka 3.4.1

[Chris Egerton]

Thanks for running this release, Luke!

On Tue, Jun 6, 2023, 22:31 Luke Chen  wrote:

> The Apache Kafka community is pleased to announce the release for
> Apache Kafka 3.4.1.
>
> This is a bug fix release and it includes fixes and improvements from
> 58 JIRAs, including a few critical bugs:
> - core
> KAFKA-14644 Process should stop after failure in raft IO thread
> KAFKA-14946 KRaft controller node shutting down while renouncing leadership
> KAFKA-14887 ZK session timeout can cause broker to shutdown
> - client
> KAFKA-14639 Kafka CooperativeStickyAssignor revokes/assigns partition
> in one rebalance cycle
> - connect
> KAFKA-12558 MM2 may not sync partition offsets correctly
> KAFKA-14666 MM2 should translate consumer group offsets behind replication
> flow
> - stream
> KAFKA-14172 bug: State stores lose state when tasks are reassigned under
> EOS
>
> All of the changes in this release can be found in the release notes:
>
> https://www.apache.org/dist/kafka/3.4.1/RELEASE_NOTES.html
>
> You can download the source and binary release (Scala 2.12 and Scala 2.13)
> from:
>
> https://kafka.apache.org/downloads#3.4.1
>
>
> ---
>
> Apache Kafka is a distributed streaming platform with four core APIs:
>
> ** The Producer API allows an application to publish a stream records
> to one or more Kafka topics.
>
> ** The Consumer API allows an application to subscribe to one or more
> topics and process the stream of records produced to them.
>
> ** The Streams API allows an application to act as a stream processor,
> consuming an input stream from one or more topics and producing an
> output stream to one or more output topics, effectively transforming
> the input streams to output streams.
>
> ** The Connector API allows building and running reusable producers or
> consumers that connect Kafka topics to existing applications or data
> systems. For example, a connector to a relational database might
> capture every change to a table.
>
>
> With these APIs, Kafka can be used for two broad classes of application:
>
> ** Building real-time streaming data pipelines that reliably get data
> between systems or applications.
>
> ** Building real-time streaming applications that transform or react
> to the streams of data.
>
> Apache Kafka is in use at large and small companies worldwide,
> including Capital One, Goldman Sachs, ING, LinkedIn, Netflix,
> Pinterest, Rabobank, Target, The New York Times, Uber, Yelp, and
> Zalando, among others.
>
> A big thank you for the following 32 contributors to this release!
>
> atu-sharm, Chia-Ping Tsai, Chris Egerton, Colin Patrick McCabe,
> csolidum, David Arthur, David Jacot, Divij Vaidya, egyedt,
> emilnkrastev, Eric Haag, Greg Harris, Guozhang Wang, Hector Geraldino,
> hudeqi, Jason Gustafson, Jeff Kim, Jorge Esteban Quilcate Otoya, José
> Armando García Sancio, Lucia Cerchie, Luke Chen, Manikumar Reddy,
> Matthias J. Sax, Mickael Maison, Philip Nee, Purshotam Chauhan, Rajini
> Sivaram, Ron Dagostino, Terry, Victoria Xia, Viktor Somogyi-Vass, Yash
> Mayya
>
> We welcome your help and feedback. For more information on how to
> report problems, and to get involved, visit the project website at
> https://kafka.apache.org/
>
>
> Thank you!
>
> Regards,
> Luke
>

[ANNOUNCE] Apache Kafka 3.4.1

[Luke Chen]

The Apache Kafka community is pleased to announce the release for
Apache Kafka 3.4.1.

This is a bug fix release and it includes fixes and improvements from
58 JIRAs, including a few critical bugs:
- core
KAFKA-14644 Process should stop after failure in raft IO thread
KAFKA-14946 KRaft controller node shutting down while renouncing leadership
KAFKA-14887 ZK session timeout can cause broker to shutdown
- client
KAFKA-14639 Kafka CooperativeStickyAssignor revokes/assigns partition
in one rebalance cycle
- connect
KAFKA-12558 MM2 may not sync partition offsets correctly
KAFKA-14666 MM2 should translate consumer group offsets behind replication flow
- stream
KAFKA-14172 bug: State stores lose state when tasks are reassigned under EOS

All of the changes in this release can be found in the release notes:

https://www.apache.org/dist/kafka/3.4.1/RELEASE_NOTES.html

You can download the source and binary release (Scala 2.12 and Scala 2.13) from:

https://kafka.apache.org/downloads#3.4.1

---

Apache Kafka is a distributed streaming platform with four core APIs:

** The Producer API allows an application to publish a stream records
to one or more Kafka topics.

** The Consumer API allows an application to subscribe to one or more
topics and process the stream of records produced to them.

** The Streams API allows an application to act as a stream processor,
consuming an input stream from one or more topics and producing an
output stream to one or more output topics, effectively transforming
the input streams to output streams.

** The Connector API allows building and running reusable producers or
consumers that connect Kafka topics to existing applications or data
systems. For example, a connector to a relational database might
capture every change to a table.


With these APIs, Kafka can be used for two broad classes of application:

** Building real-time streaming data pipelines that reliably get data
between systems or applications.

** Building real-time streaming applications that transform or react
to the streams of data.

Apache Kafka is in use at large and small companies worldwide,
including Capital One, Goldman Sachs, ING, LinkedIn, Netflix,
Pinterest, Rabobank, Target, The New York Times, Uber, Yelp, and
Zalando, among others.

A big thank you for the following 32 contributors to this release!

atu-sharm, Chia-Ping Tsai, Chris Egerton, Colin Patrick McCabe,
csolidum, David Arthur, David Jacot, Divij Vaidya, egyedt,
emilnkrastev, Eric Haag, Greg Harris, Guozhang Wang, Hector Geraldino,
hudeqi, Jason Gustafson, Jeff Kim, Jorge Esteban Quilcate Otoya, José
Armando García Sancio, Lucia Cerchie, Luke Chen, Manikumar Reddy,
Matthias J. Sax, Mickael Maison, Philip Nee, Purshotam Chauhan, Rajini
Sivaram, Ron Dagostino, Terry, Victoria Xia, Viktor Somogyi-Vass, Yash
Mayya

We welcome your help and feedback. For more information on how to
report problems, and to get involved, visit the project website at
https://kafka.apache.org/


Thank you!

Regards,
Luke

[GitHub] [kafka-site] showuon merged pull request #519: MINOR: Add Kafka 3.4.1 documentation and javadoc and download link

[via GitHub]

showuon merged PR #519:
URL: https://github.com/apache/kafka-site/pull/519


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #1904

[Apache Jenkins Server]

See 

Re: [DISCUSS] Adding non-committers as Github collaborators

[Hao Li]

Thanks John for looking into this!

Hao

On Tue, Jun 6, 2023 at 8:32 AM John Roesler  wrote:

> Hello all,
>
> I put in a ticket with Apache Infra to re-send these invites, and they
> told me I should just remove the usernames in one commit and then re-add
> them in a subsequent commit to trigger the invites again.
>
> I will go ahead and do this for the users who requested it on this thread
> (Greg and Andrew, as well as for Victoria who asked me about it
> separately). If there is anyone else who needs a re-send, please let us
> know.
>
> I'm sorry for the confusion, Hao. The docs claimed we could add 20 users,
> but when I actually checked in the file, I got an automated notification
> that we could actually only have 10.
>
> As for triggering the build, I don't believe you'll be able to log in to
> Jenkins, but you should be able to say "retest this please" in a PR comment
> to trigger it. Apparently, that doesn't work anymore, though. I'll file an
> Infra ticket for it.
>
> Thanks all,
> John
>
> On Fri, Jun 2, 2023, at 18:46, Hao Li wrote:
> > Hi Luke,
> >
> > Sorry for the late reply. Can you also add me to the whitelist? I believe
> > I'm supposed to be there as well. Matthias and John can vouch for me :)
> My
> > github ID is "lihaosky".
> >
> > Thanks,
> > Hao
> >
> > On Fri, Jun 2, 2023 at 4:43 PM Greg Harris  >
> > wrote:
> >
> >> Luke,
> >>
> >> I see that the PR has been merged, but I don't believe it re-sent the
> >> invitation.
> >>
> >> Thanks
> >> Greg
> >>
> >>
> >> On Wed, May 31, 2023 at 6:46 PM Luke Chen  wrote:
> >> >
> >> > Hi Greg and Andrew,
> >> >
> >> > Sorry, I don't know how to re-sent the invitation.
> >> > It looks like it is auto sent after the .asf.yaml updated.
> >> > Since updating collaborator list is part of release process based on
> the
> >> doc
> >> > , I just created a new list
> and
> >> > opened a PR:
> >> > https://github.com/apache/kafka/pull/13790
> >> >
> >> > Hope that after this PR merged, you'll get a new invitation.
> >> >
> >> > Thanks.
> >> > Luke
> >> >
> >> > On Thu, Jun 1, 2023 at 5:27 AM Andrew Grant
>  >> >
> >> > wrote:
> >> >
> >> > > Hi all,
> >> > > Like Greg I also received an invitation to collaborate but was too
> >> slow to
> >> > > accept the invite :( I'm also wondering if there's a way to resend
> the
> >> > > invite? I'm andymg3 on GitHub.
> >> > > Thanks,
> >> > > Andrew
> >> > >
> >> > > On Tue, May 30, 2023 at 12:12 PM Greg Harris
> >>  >> > > >
> >> > > wrote:
> >> > >
> >> > > > Hey all,
> >> > > >
> >> > > > I received an invitation to collaborate on apache/kafka, but let
> the
> >> > > > invitation expire after 7 days.
> >> > > > Is there a workflow for refreshing the invite, or is an admin
> able to
> >> > > > manually re-invite me?
> >> > > > I'm gharris1727 on github.
> >> > > >
> >> > > > Thanks!
> >> > > > Greg
> >> > > >
> >> > > > On Wed, May 24, 2023 at 9:32 AM Justine Olshan
> >> > > >  wrote:
> >> > > > >
> >> > > > > Hey Yash,
> >> > > > > I'm not sure how it used to be for sure, but I do remember we
> used
> >> to
> >> > > > have
> >> > > > > a different build system. I wonder if this used to work with the
> >> old
> >> > > > build
> >> > > > > system and not any more.
> >> > > > > I'd be curious if other projects have something similar and how
> it
> >> > > works.
> >> > > > >
> >> > > > > Thanks,
> >> > > > > Justine
> >> > > > >
> >> > > > > On Wed, May 24, 2023 at 9:22 AM Yash Mayya <
> yash.ma...@gmail.com>
> >> > > wrote:
> >> > > > >
> >> > > > > > Hi Justine,
> >> > > > > >
> >> > > > > > Thanks for the response. Non-committers don't have Apache
> >> accounts;
> >> > > > are you
> >> > > > > > suggesting that there wasn't a need to sign in earlier and a
> >> change
> >> > > in
> >> > > > this
> >> > > > > > policy is restricting collaborators from triggering Jenkins
> >> builds?
> >> > > > > >
> >> > > > > > Thanks,
> >> > > > > > Yash
> >> > > > > >
> >> > > > > > On Wed, May 24, 2023 at 9:30 PM Justine Olshan
> >> > > > > > 
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > > > Yash,
> >> > > > > > >
> >> > > > > > > When I rebuild, I go to the CloudBees CI page and I have to
> >> log in
> >> > > > with
> >> > > > > > my
> >> > > > > > > apache account.
> >> > > > > > > Not sure if the change in the build system or the need to
> sign
> >> in
> >> > > is
> >> > > > part
> >> > > > > > > of the problem.
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > On Wed, May 24, 2023 at 4:54 AM Federico Valeri <
> >> > > > fedeval...@gmail.com>
> >> > > > > > > wrote:
> >> > > > > > >
> >> > > > > > > > +1 on Divij suggestions
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > On Wed, May 24, 2023 at 12:04 PM Divij Vaidya <
> >> > > > divijvaidy...@gmail.com
> >> > > > > > >
> >> > > > > > > > wrote:
> >> > > > > > > > >
> >> > > > > > > > > Hey folks
> >> > > > > > > > >
> >> > > > > > > > > A week into this experiment, I am finding the ability to
> >> add
> >> > > > 

Re: [DISCUSS] Adding non-committers as Github collaborators

[Andrew Grant]

Got the invite and was able to accept.

Thanks a lot John!

On Tue, Jun 6, 2023 at 6:11 PM Greg Harris 
wrote:

> John,
>
> I saw the invite be re-sent, and accepted it.
>
> Thanks for following up!
> Greg
>
> On Tue, Jun 6, 2023 at 3:08 PM John Roesler  wrote:
> >
> > Hello Greg, Andrew, and Victoria,
> >
> > I have just re-added you to the file, so you should receive new invites.
> Please let us know if you have any trouble!
> >
> > Thanks,
> > -John
> >
> > On Tue, Jun 6, 2023, at 10:31, John Roesler wrote:
> > > Hello all,
> > >
> > > I put in a ticket with Apache Infra to re-send these invites, and they
> > > told me I should just remove the usernames in one commit and then
> > > re-add them in a subsequent commit to trigger the invites again.
> > >
> > > I will go ahead and do this for the users who requested it on this
> > > thread (Greg and Andrew, as well as for Victoria who asked me about it
> > > separately). If there is anyone else who needs a re-send, please let us
> > > know.
> > >
> > > I'm sorry for the confusion, Hao. The docs claimed we could add 20
> > > users, but when I actually checked in the file, I got an automated
> > > notification that we could actually only have 10.
> > >
> > > As for triggering the build, I don't believe you'll be able to log in
> > > to Jenkins, but you should be able to say "retest this please" in a PR
> > > comment to trigger it. Apparently, that doesn't work anymore, though.
> > > I'll file an Infra ticket for it.
> > >
> > > Thanks all,
> > > John
> > >
> > > On Fri, Jun 2, 2023, at 18:46, Hao Li wrote:
> > >> Hi Luke,
> > >>
> > >> Sorry for the late reply. Can you also add me to the whitelist? I
> believe
> > >> I'm supposed to be there as well. Matthias and John can vouch for me
> :) My
> > >> github ID is "lihaosky".
> > >>
> > >> Thanks,
> > >> Hao
> > >>
> > >> On Fri, Jun 2, 2023 at 4:43 PM Greg Harris
> 
> > >> wrote:
> > >>
> > >>> Luke,
> > >>>
> > >>> I see that the PR has been merged, but I don't believe it re-sent the
> > >>> invitation.
> > >>>
> > >>> Thanks
> > >>> Greg
> > >>>
> > >>>
> > >>> On Wed, May 31, 2023 at 6:46 PM Luke Chen  wrote:
> > >>> >
> > >>> > Hi Greg and Andrew,
> > >>> >
> > >>> > Sorry, I don't know how to re-sent the invitation.
> > >>> > It looks like it is auto sent after the .asf.yaml updated.
> > >>> > Since updating collaborator list is part of release process based
> on the
> > >>> doc
> > >>> > , I just created a new
> list and
> > >>> > opened a PR:
> > >>> > https://github.com/apache/kafka/pull/13790
> > >>> >
> > >>> > Hope that after this PR merged, you'll get a new invitation.
> > >>> >
> > >>> > Thanks.
> > >>> > Luke
> > >>> >
> > >>> > On Thu, Jun 1, 2023 at 5:27 AM Andrew Grant
>  > >>> >
> > >>> > wrote:
> > >>> >
> > >>> > > Hi all,
> > >>> > > Like Greg I also received an invitation to collaborate but was
> too
> > >>> slow to
> > >>> > > accept the invite :( I'm also wondering if there's a way to
> resend the
> > >>> > > invite? I'm andymg3 on GitHub.
> > >>> > > Thanks,
> > >>> > > Andrew
> > >>> > >
> > >>> > > On Tue, May 30, 2023 at 12:12 PM Greg Harris
> > >>>  > >>> > > >
> > >>> > > wrote:
> > >>> > >
> > >>> > > > Hey all,
> > >>> > > >
> > >>> > > > I received an invitation to collaborate on apache/kafka, but
> let the
> > >>> > > > invitation expire after 7 days.
> > >>> > > > Is there a workflow for refreshing the invite, or is an admin
> able to
> > >>> > > > manually re-invite me?
> > >>> > > > I'm gharris1727 on github.
> > >>> > > >
> > >>> > > > Thanks!
> > >>> > > > Greg
> > >>> > > >
> > >>> > > > On Wed, May 24, 2023 at 9:32 AM Justine Olshan
> > >>> > > >  wrote:
> > >>> > > > >
> > >>> > > > > Hey Yash,
> > >>> > > > > I'm not sure how it used to be for sure, but I do remember
> we used
> > >>> to
> > >>> > > > have
> > >>> > > > > a different build system. I wonder if this used to work with
> the
> > >>> old
> > >>> > > > build
> > >>> > > > > system and not any more.
> > >>> > > > > I'd be curious if other projects have something similar and
> how it
> > >>> > > works.
> > >>> > > > >
> > >>> > > > > Thanks,
> > >>> > > > > Justine
> > >>> > > > >
> > >>> > > > > On Wed, May 24, 2023 at 9:22 AM Yash Mayya <
> yash.ma...@gmail.com>
> > >>> > > wrote:
> > >>> > > > >
> > >>> > > > > > Hi Justine,
> > >>> > > > > >
> > >>> > > > > > Thanks for the response. Non-committers don't have Apache
> > >>> accounts;
> > >>> > > > are you
> > >>> > > > > > suggesting that there wasn't a need to sign in earlier and
> a
> > >>> change
> > >>> > > in
> > >>> > > > this
> > >>> > > > > > policy is restricting collaborators from triggering Jenkins
> > >>> builds?
> > >>> > > > > >
> > >>> > > > > > Thanks,
> > >>> > > > > > Yash
> > >>> > > > > >
> > >>> > > > > > On Wed, May 24, 2023 at 9:30 PM Justine Olshan
> > >>> > > > > > 
> > >>> > > > > > wrote:
> > >>> > > > > >
> > >>> > > > > > > Yash,
> > >>> > > > > > >
> > >>> > > > > > > When I rebuild, I 

Re: [DISCUSS] Adding non-committers as Github collaborators

[Greg Harris]

John,

I saw the invite be re-sent, and accepted it.

Thanks for following up!
Greg

On Tue, Jun 6, 2023 at 3:08 PM John Roesler  wrote:
>
> Hello Greg, Andrew, and Victoria,
>
> I have just re-added you to the file, so you should receive new invites. 
> Please let us know if you have any trouble!
>
> Thanks,
> -John
>
> On Tue, Jun 6, 2023, at 10:31, John Roesler wrote:
> > Hello all,
> >
> > I put in a ticket with Apache Infra to re-send these invites, and they
> > told me I should just remove the usernames in one commit and then
> > re-add them in a subsequent commit to trigger the invites again.
> >
> > I will go ahead and do this for the users who requested it on this
> > thread (Greg and Andrew, as well as for Victoria who asked me about it
> > separately). If there is anyone else who needs a re-send, please let us
> > know.
> >
> > I'm sorry for the confusion, Hao. The docs claimed we could add 20
> > users, but when I actually checked in the file, I got an automated
> > notification that we could actually only have 10.
> >
> > As for triggering the build, I don't believe you'll be able to log in
> > to Jenkins, but you should be able to say "retest this please" in a PR
> > comment to trigger it. Apparently, that doesn't work anymore, though.
> > I'll file an Infra ticket for it.
> >
> > Thanks all,
> > John
> >
> > On Fri, Jun 2, 2023, at 18:46, Hao Li wrote:
> >> Hi Luke,
> >>
> >> Sorry for the late reply. Can you also add me to the whitelist? I believe
> >> I'm supposed to be there as well. Matthias and John can vouch for me :) My
> >> github ID is "lihaosky".
> >>
> >> Thanks,
> >> Hao
> >>
> >> On Fri, Jun 2, 2023 at 4:43 PM Greg Harris 
> >> wrote:
> >>
> >>> Luke,
> >>>
> >>> I see that the PR has been merged, but I don't believe it re-sent the
> >>> invitation.
> >>>
> >>> Thanks
> >>> Greg
> >>>
> >>>
> >>> On Wed, May 31, 2023 at 6:46 PM Luke Chen  wrote:
> >>> >
> >>> > Hi Greg and Andrew,
> >>> >
> >>> > Sorry, I don't know how to re-sent the invitation.
> >>> > It looks like it is auto sent after the .asf.yaml updated.
> >>> > Since updating collaborator list is part of release process based on the
> >>> doc
> >>> > , I just created a new list and
> >>> > opened a PR:
> >>> > https://github.com/apache/kafka/pull/13790
> >>> >
> >>> > Hope that after this PR merged, you'll get a new invitation.
> >>> >
> >>> > Thanks.
> >>> > Luke
> >>> >
> >>> > On Thu, Jun 1, 2023 at 5:27 AM Andrew Grant  >>> >
> >>> > wrote:
> >>> >
> >>> > > Hi all,
> >>> > > Like Greg I also received an invitation to collaborate but was too
> >>> slow to
> >>> > > accept the invite :( I'm also wondering if there's a way to resend the
> >>> > > invite? I'm andymg3 on GitHub.
> >>> > > Thanks,
> >>> > > Andrew
> >>> > >
> >>> > > On Tue, May 30, 2023 at 12:12 PM Greg Harris
> >>>  >>> > > >
> >>> > > wrote:
> >>> > >
> >>> > > > Hey all,
> >>> > > >
> >>> > > > I received an invitation to collaborate on apache/kafka, but let the
> >>> > > > invitation expire after 7 days.
> >>> > > > Is there a workflow for refreshing the invite, or is an admin able 
> >>> > > > to
> >>> > > > manually re-invite me?
> >>> > > > I'm gharris1727 on github.
> >>> > > >
> >>> > > > Thanks!
> >>> > > > Greg
> >>> > > >
> >>> > > > On Wed, May 24, 2023 at 9:32 AM Justine Olshan
> >>> > > >  wrote:
> >>> > > > >
> >>> > > > > Hey Yash,
> >>> > > > > I'm not sure how it used to be for sure, but I do remember we used
> >>> to
> >>> > > > have
> >>> > > > > a different build system. I wonder if this used to work with the
> >>> old
> >>> > > > build
> >>> > > > > system and not any more.
> >>> > > > > I'd be curious if other projects have something similar and how it
> >>> > > works.
> >>> > > > >
> >>> > > > > Thanks,
> >>> > > > > Justine
> >>> > > > >
> >>> > > > > On Wed, May 24, 2023 at 9:22 AM Yash Mayya 
> >>> > > wrote:
> >>> > > > >
> >>> > > > > > Hi Justine,
> >>> > > > > >
> >>> > > > > > Thanks for the response. Non-committers don't have Apache
> >>> accounts;
> >>> > > > are you
> >>> > > > > > suggesting that there wasn't a need to sign in earlier and a
> >>> change
> >>> > > in
> >>> > > > this
> >>> > > > > > policy is restricting collaborators from triggering Jenkins
> >>> builds?
> >>> > > > > >
> >>> > > > > > Thanks,
> >>> > > > > > Yash
> >>> > > > > >
> >>> > > > > > On Wed, May 24, 2023 at 9:30 PM Justine Olshan
> >>> > > > > > 
> >>> > > > > > wrote:
> >>> > > > > >
> >>> > > > > > > Yash,
> >>> > > > > > >
> >>> > > > > > > When I rebuild, I go to the CloudBees CI page and I have to
> >>> log in
> >>> > > > with
> >>> > > > > > my
> >>> > > > > > > apache account.
> >>> > > > > > > Not sure if the change in the build system or the need to sign
> >>> in
> >>> > > is
> >>> > > > part
> >>> > > > > > > of the problem.
> >>> > > > > > >
> >>> > > > > > >
> >>> > > > > > > On Wed, May 24, 2023 at 4:54 AM Federico Valeri <
> >>> > > > fedeval...@gmail.com>
> >>> > > > > > > wrote:
> 

Re: [DISCUSS] Adding non-committers as Github collaborators

[John Roesler]

Hello Greg, Andrew, and Victoria,

I have just re-added you to the file, so you should receive new invites. Please 
let us know if you have any trouble!

Thanks,
-John

On Tue, Jun 6, 2023, at 10:31, John Roesler wrote:
> Hello all,
>
> I put in a ticket with Apache Infra to re-send these invites, and they 
> told me I should just remove the usernames in one commit and then 
> re-add them in a subsequent commit to trigger the invites again.
>
> I will go ahead and do this for the users who requested it on this 
> thread (Greg and Andrew, as well as for Victoria who asked me about it 
> separately). If there is anyone else who needs a re-send, please let us 
> know.
>
> I'm sorry for the confusion, Hao. The docs claimed we could add 20 
> users, but when I actually checked in the file, I got an automated 
> notification that we could actually only have 10.
>
> As for triggering the build, I don't believe you'll be able to log in 
> to Jenkins, but you should be able to say "retest this please" in a PR 
> comment to trigger it. Apparently, that doesn't work anymore, though. 
> I'll file an Infra ticket for it.
>
> Thanks all,
> John
>
> On Fri, Jun 2, 2023, at 18:46, Hao Li wrote:
>> Hi Luke,
>>
>> Sorry for the late reply. Can you also add me to the whitelist? I believe
>> I'm supposed to be there as well. Matthias and John can vouch for me :) My
>> github ID is "lihaosky".
>>
>> Thanks,
>> Hao
>>
>> On Fri, Jun 2, 2023 at 4:43 PM Greg Harris 
>> wrote:
>>
>>> Luke,
>>>
>>> I see that the PR has been merged, but I don't believe it re-sent the
>>> invitation.
>>>
>>> Thanks
>>> Greg
>>>
>>>
>>> On Wed, May 31, 2023 at 6:46 PM Luke Chen  wrote:
>>> >
>>> > Hi Greg and Andrew,
>>> >
>>> > Sorry, I don't know how to re-sent the invitation.
>>> > It looks like it is auto sent after the .asf.yaml updated.
>>> > Since updating collaborator list is part of release process based on the
>>> doc
>>> > , I just created a new list and
>>> > opened a PR:
>>> > https://github.com/apache/kafka/pull/13790
>>> >
>>> > Hope that after this PR merged, you'll get a new invitation.
>>> >
>>> > Thanks.
>>> > Luke
>>> >
>>> > On Thu, Jun 1, 2023 at 5:27 AM Andrew Grant >> >
>>> > wrote:
>>> >
>>> > > Hi all,
>>> > > Like Greg I also received an invitation to collaborate but was too
>>> slow to
>>> > > accept the invite :( I'm also wondering if there's a way to resend the
>>> > > invite? I'm andymg3 on GitHub.
>>> > > Thanks,
>>> > > Andrew
>>> > >
>>> > > On Tue, May 30, 2023 at 12:12 PM Greg Harris
>>> >> > > >
>>> > > wrote:
>>> > >
>>> > > > Hey all,
>>> > > >
>>> > > > I received an invitation to collaborate on apache/kafka, but let the
>>> > > > invitation expire after 7 days.
>>> > > > Is there a workflow for refreshing the invite, or is an admin able to
>>> > > > manually re-invite me?
>>> > > > I'm gharris1727 on github.
>>> > > >
>>> > > > Thanks!
>>> > > > Greg
>>> > > >
>>> > > > On Wed, May 24, 2023 at 9:32 AM Justine Olshan
>>> > > >  wrote:
>>> > > > >
>>> > > > > Hey Yash,
>>> > > > > I'm not sure how it used to be for sure, but I do remember we used
>>> to
>>> > > > have
>>> > > > > a different build system. I wonder if this used to work with the
>>> old
>>> > > > build
>>> > > > > system and not any more.
>>> > > > > I'd be curious if other projects have something similar and how it
>>> > > works.
>>> > > > >
>>> > > > > Thanks,
>>> > > > > Justine
>>> > > > >
>>> > > > > On Wed, May 24, 2023 at 9:22 AM Yash Mayya 
>>> > > wrote:
>>> > > > >
>>> > > > > > Hi Justine,
>>> > > > > >
>>> > > > > > Thanks for the response. Non-committers don't have Apache
>>> accounts;
>>> > > > are you
>>> > > > > > suggesting that there wasn't a need to sign in earlier and a
>>> change
>>> > > in
>>> > > > this
>>> > > > > > policy is restricting collaborators from triggering Jenkins
>>> builds?
>>> > > > > >
>>> > > > > > Thanks,
>>> > > > > > Yash
>>> > > > > >
>>> > > > > > On Wed, May 24, 2023 at 9:30 PM Justine Olshan
>>> > > > > > 
>>> > > > > > wrote:
>>> > > > > >
>>> > > > > > > Yash,
>>> > > > > > >
>>> > > > > > > When I rebuild, I go to the CloudBees CI page and I have to
>>> log in
>>> > > > with
>>> > > > > > my
>>> > > > > > > apache account.
>>> > > > > > > Not sure if the change in the build system or the need to sign
>>> in
>>> > > is
>>> > > > part
>>> > > > > > > of the problem.
>>> > > > > > >
>>> > > > > > >
>>> > > > > > > On Wed, May 24, 2023 at 4:54 AM Federico Valeri <
>>> > > > fedeval...@gmail.com>
>>> > > > > > > wrote:
>>> > > > > > >
>>> > > > > > > > +1 on Divij suggestions
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > > On Wed, May 24, 2023 at 12:04 PM Divij Vaidya <
>>> > > > divijvaidy...@gmail.com
>>> > > > > > >
>>> > > > > > > > wrote:
>>> > > > > > > > >
>>> > > > > > > > > Hey folks
>>> > > > > > > > >
>>> > > > > > > > > A week into this experiment, I am finding the ability to
>>> add
>>> > > > labels,
>>> > > > > > > > > request for reviewers and 

Re: [DISCUSS] Adding non-committers as Github collaborators

[John Roesler]

Hi Divij,

Interesting; perhaps they didn't need to send you an invite because your github 
account is already associated with another Apache project? The actions you 
listed are the things that the collaborator role gives you, so I think you're 
good to go.

Thanks,
-John

On Tue, Jun 6, 2023, at 11:14, Divij Vaidya wrote:
> Hey John
>
> What does the invite do? I did not receive any invite either but I have
> been able to add labels, close PRs etc. since I was added to the
> contributors list (perhaps because I already have an apache LDAP account?).
> I am working with the assumption that things are working for me even if I
> didn't have to accept an invite. Is that a correct assumption?
>
> --
> Divij Vaidya
>
>
>
> On Tue, Jun 6, 2023 at 5:32 PM John Roesler  wrote:
>
>> Hello all,
>>
>> I put in a ticket with Apache Infra to re-send these invites, and they
>> told me I should just remove the usernames in one commit and then re-add
>> them in a subsequent commit to trigger the invites again.
>>
>> I will go ahead and do this for the users who requested it on this thread
>> (Greg and Andrew, as well as for Victoria who asked me about it
>> separately). If there is anyone else who needs a re-send, please let us
>> know.
>>
>> I'm sorry for the confusion, Hao. The docs claimed we could add 20 users,
>> but when I actually checked in the file, I got an automated notification
>> that we could actually only have 10.
>>
>> As for triggering the build, I don't believe you'll be able to log in to
>> Jenkins, but you should be able to say "retest this please" in a PR comment
>> to trigger it. Apparently, that doesn't work anymore, though. I'll file an
>> Infra ticket for it.
>>
>> Thanks all,
>> John
>>
>> On Fri, Jun 2, 2023, at 18:46, Hao Li wrote:
>> > Hi Luke,
>> >
>> > Sorry for the late reply. Can you also add me to the whitelist? I believe
>> > I'm supposed to be there as well. Matthias and John can vouch for me :)
>> My
>> > github ID is "lihaosky".
>> >
>> > Thanks,
>> > Hao
>> >
>> > On Fri, Jun 2, 2023 at 4:43 PM Greg Harris > >
>> > wrote:
>> >
>> >> Luke,
>> >>
>> >> I see that the PR has been merged, but I don't believe it re-sent the
>> >> invitation.
>> >>
>> >> Thanks
>> >> Greg
>> >>
>> >>
>> >> On Wed, May 31, 2023 at 6:46 PM Luke Chen  wrote:
>> >> >
>> >> > Hi Greg and Andrew,
>> >> >
>> >> > Sorry, I don't know how to re-sent the invitation.
>> >> > It looks like it is auto sent after the .asf.yaml updated.
>> >> > Since updating collaborator list is part of release process based on
>> the
>> >> doc
>> >> > , I just created a new list
>> and
>> >> > opened a PR:
>> >> > https://github.com/apache/kafka/pull/13790
>> >> >
>> >> > Hope that after this PR merged, you'll get a new invitation.
>> >> >
>> >> > Thanks.
>> >> > Luke
>> >> >
>> >> > On Thu, Jun 1, 2023 at 5:27 AM Andrew Grant
>> > >> >
>> >> > wrote:
>> >> >
>> >> > > Hi all,
>> >> > > Like Greg I also received an invitation to collaborate but was too
>> >> slow to
>> >> > > accept the invite :( I'm also wondering if there's a way to resend
>> the
>> >> > > invite? I'm andymg3 on GitHub.
>> >> > > Thanks,
>> >> > > Andrew
>> >> > >
>> >> > > On Tue, May 30, 2023 at 12:12 PM Greg Harris
>> >> > >> > > >
>> >> > > wrote:
>> >> > >
>> >> > > > Hey all,
>> >> > > >
>> >> > > > I received an invitation to collaborate on apache/kafka, but let
>> the
>> >> > > > invitation expire after 7 days.
>> >> > > > Is there a workflow for refreshing the invite, or is an admin
>> able to
>> >> > > > manually re-invite me?
>> >> > > > I'm gharris1727 on github.
>> >> > > >
>> >> > > > Thanks!
>> >> > > > Greg
>> >> > > >
>> >> > > > On Wed, May 24, 2023 at 9:32 AM Justine Olshan
>> >> > > >  wrote:
>> >> > > > >
>> >> > > > > Hey Yash,
>> >> > > > > I'm not sure how it used to be for sure, but I do remember we
>> used
>> >> to
>> >> > > > have
>> >> > > > > a different build system. I wonder if this used to work with the
>> >> old
>> >> > > > build
>> >> > > > > system and not any more.
>> >> > > > > I'd be curious if other projects have something similar and how
>> it
>> >> > > works.
>> >> > > > >
>> >> > > > > Thanks,
>> >> > > > > Justine
>> >> > > > >
>> >> > > > > On Wed, May 24, 2023 at 9:22 AM Yash Mayya <
>> yash.ma...@gmail.com>
>> >> > > wrote:
>> >> > > > >
>> >> > > > > > Hi Justine,
>> >> > > > > >
>> >> > > > > > Thanks for the response. Non-committers don't have Apache
>> >> accounts;
>> >> > > > are you
>> >> > > > > > suggesting that there wasn't a need to sign in earlier and a
>> >> change
>> >> > > in
>> >> > > > this
>> >> > > > > > policy is restricting collaborators from triggering Jenkins
>> >> builds?
>> >> > > > > >
>> >> > > > > > Thanks,
>> >> > > > > > Yash
>> >> > > > > >
>> >> > > > > > On Wed, May 24, 2023 at 9:30 PM Justine Olshan
>> >> > > > > > 
>> >> > > > > > wrote:
>> >> > > > > >
>> >> > > > > > > Yash,
>> >> > > > > > >
>> >> > > > > > > When I rebuild, I go to the 

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1903

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 575522 lines...]
[2023-06-06T21:54:42.133Z] 
[2023-06-06T21:54:42.133Z] > Task :streams:javadoc
[2023-06-06T21:54:43.061Z] > Task :streams:javadocJar
[2023-06-06T21:54:43.989Z] 
[2023-06-06T21:54:43.989Z] > Task :clients:javadoc
[2023-06-06T21:54:43.989Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk@2/clients/src/main/java/org/apache/kafka/clients/admin/ScramMechanism.java:32:
 warning - Tag @see: missing final '>': "https://cwiki.apache.org/confluence/display/KAFKA/KIP-554%3A+Add+Broker-side+SCRAM+Config+API;>KIP-554:
 Add Broker-side SCRAM Config API
[2023-06-06T21:54:43.989Z] 
[2023-06-06T21:54:43.989Z]  This code is duplicated in 
org.apache.kafka.common.security.scram.internals.ScramMechanism.
[2023-06-06T21:54:43.989Z]  The type field in both files must match and must 
not change. The type field
[2023-06-06T21:54:43.989Z]  is used both for passing ScramCredentialUpsertion 
and for the internal
[2023-06-06T21:54:43.989Z]  UserScramCredentialRecord. Do not change the type 
field."
[2023-06-06T21:54:45.728Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk@2/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/secured/package-info.java:21:
 warning - Tag @link: reference not found: 
org.apache.kafka.common.security.oauthbearer
[2023-06-06T21:54:45.728Z] 5 warnings
[2023-06-06T21:54:47.467Z] 
[2023-06-06T21:54:47.467Z] > Task :clients:javadocJar
[2023-06-06T21:54:48.395Z] > Task :clients:srcJar
[2023-06-06T21:54:49.324Z] > Task :clients:testJar
[2023-06-06T21:54:49.324Z] > Task :clients:testSrcJar
[2023-06-06T21:54:49.324Z] > Task 
:clients:publishMavenJavaPublicationToMavenLocal
[2023-06-06T21:54:49.324Z] > Task :clients:publishToMavenLocal
[2023-06-06T21:55:05.634Z] > Task :core:compileScala
[2023-06-06T21:56:55.040Z] > Task :core:classes
[2023-06-06T21:56:55.040Z] > Task :core:compileTestJava NO-SOURCE
[2023-06-06T21:57:13.911Z] > Task :core:compileTestScala
[2023-06-06T21:58:47.420Z] > Task :core:testClasses
[2023-06-06T21:58:47.420Z] > Task :streams:compileTestJava UP-TO-DATE
[2023-06-06T21:58:47.420Z] > Task :streams:testClasses UP-TO-DATE
[2023-06-06T21:58:47.420Z] > Task :streams:testJar
[2023-06-06T21:58:47.420Z] > Task :streams:testSrcJar
[2023-06-06T21:58:47.420Z] > Task 
:streams:publishMavenJavaPublicationToMavenLocal
[2023-06-06T21:58:47.420Z] > Task :streams:publishToMavenLocal
[2023-06-06T21:58:47.420Z] 
[2023-06-06T21:58:47.420Z] Deprecated Gradle features were used in this build, 
making it incompatible with Gradle 9.0.
[2023-06-06T21:58:47.420Z] 
[2023-06-06T21:58:47.420Z] You can use '--warning-mode all' to show the 
individual deprecation warnings and determine if they come from your own 
scripts or plugins.
[2023-06-06T21:58:47.420Z] 
[2023-06-06T21:58:47.420Z] See 
https://docs.gradle.org/8.1.1/userguide/command_line_interface.html#sec:command_line_warnings
[2023-06-06T21:58:47.420Z] 
[2023-06-06T21:58:47.420Z] BUILD SUCCESSFUL in 4m 43s
[2023-06-06T21:58:47.420Z] 89 actionable tasks: 33 executed, 56 up-to-date
[Pipeline] sh
[2023-06-06T21:58:50.051Z] + grep ^version= gradle.properties
[2023-06-06T21:58:50.051Z] + cut -d= -f 2
[Pipeline] dir
[2023-06-06T21:58:50.730Z] Running in 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk@2/streams/quickstart
[Pipeline] {
[Pipeline] sh
[2023-06-06T21:58:52.850Z] + mvn clean install -Dgpg.skip
[2023-06-06T21:58:55.622Z] [INFO] Scanning for projects...
[2023-06-06T21:58:55.622Z] [INFO] 

[2023-06-06T21:58:55.622Z] [INFO] Reactor Build Order:
[2023-06-06T21:58:55.622Z] [INFO] 
[2023-06-06T21:58:55.622Z] [INFO] Kafka Streams :: Quickstart   
 [pom]
[2023-06-06T21:58:55.622Z] [INFO] streams-quickstart-java   
 [maven-archetype]
[2023-06-06T21:58:55.622Z] [INFO] 
[2023-06-06T21:58:55.622Z] [INFO] < 
org.apache.kafka:streams-quickstart >-
[2023-06-06T21:58:55.622Z] [INFO] Building Kafka Streams :: Quickstart 
3.6.0-SNAPSHOT[1/2]
[2023-06-06T21:58:55.622Z] [INFO]   from pom.xml
[2023-06-06T21:58:55.622Z] [INFO] [ pom 
]-
[2023-06-06T21:58:55.622Z] [INFO] 
[2023-06-06T21:58:55.622Z] [INFO] --- clean:3.0.0:clean (default-clean) @ 
streams-quickstart ---
[2023-06-06T21:58:56.551Z] [INFO] 
[2023-06-06T21:58:56.551Z] [INFO] --- remote-resources:1.5:process 
(process-resource-bundles) @ streams-quickstart ---
[2023-06-06T21:58:57.478Z] [INFO] 
[2023-06-06T21:58:57.478Z] [INFO] --- site:3.5.1:attach-descriptor 
(attach-descriptor) @ streams-quickstart ---
[2023-06-06T21:58:58.407Z] [INFO] 
[2023-06-06T21:58:58.407Z] [INFO] --- gpg:1.6:sign (sign-artifacts) @ 
streams-quickstart ---
[2023-06-06T21:58:58.407Z] [INFO] 

Re: [DISCUSS] KIP-923: Add A Grace Period to Stream Table Join

[Victoria Xia]

Handling the "changelog" topic for the buffer in the same way as
repartition topics makes sense. Thanks for clearing that up!

- Victoria

On Tue, Jun 6, 2023 at 10:22 AM Walker Carlson
 wrote:

> Good Point Victoria. I just removed the compacted topic mention from the
> KIP. I agree with Burno about using a normal topic and deleting records
> that have been processed.
>
> On Tue, Jun 6, 2023 at 2:28 AM Bruno Cadonna  wrote:
>
> > Hi,
> >
> > another idea that came to my mind. Instead of using a compacted topic,
> > the buffer could use a non-compacted topic and regularly delete records
> > before a given offset as Streams does for repartition topics.
> >
> > Best,
> > Bruno
> >
> > On 05.06.23 21:48, Bruno Cadonna wrote:
> > > Hi Victoria,
> > >
> > > that is a good point!
> > >
> > > I think, the topic needs to be a compacted topic to be able to get rid
> > > of records that are evicted from the buffer. So the key might be
> > > something with the key, the timestamp, and a sequence number to
> > > distinguish between records with the same key and same timestamp.
> > >
> > > Just an idea! Maybe Walker comes up with something better.
> > >
> > > Best,
> > > Bruno
> > >
> > > On 05.06.23 20:38, Victoria Xia wrote:
> > >> Hi Walker,
> > >>
> > >> Thanks for the latest updates! The KIP looks great. Just one question
> > >> about
> > >> the changelog topic for the join buffer: The KIP says "When a failure
> > >> occurs the buffer will try to recover from an OffsetCheckpoint if
> > >> possible.
> > >> If not it will reload the buffer from a compacted change-log topic."
> > This
> > >> is a new changelog topic that will be introduced specifically for the
> > >> join
> > >> buffer, right? Why is the changelog topic compacted? What are the
> keys?
> > I
> > >> am confused because the buffer contains records from the stream-side
> > >> of the
> > >> join, for which multiple records with the same key should be treated
> as
> > >> separate updates will all must be tracked in the buffer, rather than
> > >> updates which replace each other.
> > >>
> > >> Thanks,
> > >> Victoria
> > >>
> > >> On Mon, Jun 5, 2023 at 1:47 AM Bruno Cadonna 
> > wrote:
> > >>
> > >>> Hi Walker,
> > >>>
> > >>> Thanks once more for the updates to the KIP!
> > >>>
> > >>> Do you also plan to expose metrics for the buffer?
> > >>>
> > >>> Best,
> > >>> Bruno
> > >>>
> > >>> On 02.06.23 17:16, Walker Carlson wrote:
> >  Hello Bruno,
> > 
> >  I think this covers your questions. Let me know what you think
> > 
> >  2.
> >  We can use a changelog topic. I think we can treat it like any other
> > >>> store
> >  and recover in the usual manner. Also implementation is on disk
> > 
> >  3.
> >  The description is in the public interfaces description. I will copy
> > it
> >  into the proposed changes as well.
> > 
> >  This is a bit of an implementation detail that I didn't want to add
> >  into
> >  the kip, but the record will be added to the buffer to keep the
> stream
> > >>> time
> >  consistent, it will just be ejected immediately. If of course if
> this
> >  causes performance issues we will skip this step and track stream
> time
> >  separately. I will update the kip to say that stream time advances
> >  when a
> >  stream record enters the node.
> > 
> >  Also, yes, updated.
> > 
> >  5.
> >  No there is no difference right now, everything gets processed as it
> > >>> comes
> >  in and tries to find a record for its time stamp.
> > 
> >  Walker
> > 
> >  On Fri, Jun 2, 2023 at 6:41 AM Bruno Cadonna 
> >  wrote:
> > 
> > > Hi Walker,
> > >
> > > Thanks for the updates!
> > >
> > > 2.
> > > It is still not clear to me how a failure is handled. I do not
> > > understand what you mean by "recover from an OffsetCheckpoint".
> > >
> > > My understanding is that the buffer needs to be replicated into its
> > > own
> > > Kafka topic. The input topic is not enough. The offset of a record
> is
> > > added to the offsets to commit once the record is streamed through
> > the
> > > subtopology. That means once the record is added to the buffer its
> > > offset is added to the offsets to commit -- independently of
> > > whether the
> > > record was evicted from the buffer and sent to the join node or
> not.
> > > Now, let's assume the following scenario
> > > 1. a record is read from the input topic and added to the buffer,
> but
> > > not evicted to be processed by the join node.
> > > 2. When the processing of the subtopology finishes the offset of
> the
> > > record is added to the offsets to commit.
> > > 3. A commit happens.
> > > 4. A failure happens
> > >
> > > After the failure the buffer is empty but the record will not be
> read
> > > anymore from the input topic since its offset has been already
> > > committed. 

Re: [VOTE] 3.5.0 RC1

[Jakub Scholz]

+1 (non-binding) ... I used the staged binaries with Scala 2.13 and staged
artifacts to run my tests. All seems to work fine.

Thanks for running the release Mickael!

Jakub

On Mon, Jun 5, 2023 at 3:39 PM Mickael Maison  wrote:

> Hello Kafka users, developers and client-developers,
>
> This is the second candidate for release of Apache Kafka 3.5.0. Some
> of the major features include:
> - KIP-710: Full support for distributed mode in dedicated MirrorMaker
> 2.0 clusters
> - KIP-881: Rack-aware Partition Assignment for Kafka Consumers
> - KIP-887: Add ConfigProvider to make use of environment variables
> - KIP-889: Versioned State Stores
> - KIP-894: Use incrementalAlterConfig for syncing topic configurations
> - KIP-900: KRaft kafka-storage.sh API additions to support SCRAM for
> Kafka Brokers
>
> Release notes for the 3.5.0 release:
> https://home.apache.org/~mimaison/kafka-3.5.0-rc1/RELEASE_NOTES.html
>
> *** Please download, test and vote by Friday June 9, 5pm PT
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> https://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> https://home.apache.org/~mimaison/kafka-3.5.0-rc1/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>
> * Javadoc:
> https://home.apache.org/~mimaison/kafka-3.5.0-rc1/javadoc/
>
> * Tag to be voted upon (off 3.5 branch) is the 3.5.0 tag:
> https://github.com/apache/kafka/releases/tag/3.5.0-rc1
>
> * Documentation:
> https://kafka.apache.org/35/documentation.html
>
> * Protocol:
> https://kafka.apache.org/35/protocol.html
>
> * Successful Jenkins builds for the 3.5 branch:
> Unit/integration tests: I'm struggling to get all tests to pass in the
> same build. I'll run a few more builds to ensure each test pass at
> least once in the CI. All tests passed locally.
> System tests: The build is still running, I'll send an update once I
> have the results.
>
> Thanks,
> Mickael
>

Re: [DISCUSS] Partial CI builds - Reducing flakiness with fewer tests

[Greg Harris]

David,

Thanks for your thoughts!

> Indeed, they will be more likely to pass but the
> downside is that folks may start to only rely on that signal and commit
> without looking at the full test suite. This seems dangerous to me.

I completely agree with you that it is not desirable for committers to
become overly reliant on the smaller subset of tests. Instead, the
partial builds can be a new merge requirement, instead of a
replacement for the existing full-suite spot-check.
For PRs which fail the partial test run, committers won't need to
examine the full suite to know that the contributor needs to make
further changes, and can spend their attention on something else.
We can make this clear with a dev-list announcement to explain the
meaning and interpretation of the new build result. We can also call
for flakiness reduction contributions at the same time.

> I would rather focus on trying to address this first. If
> we can stabilize them, I wonder if we should also enforce a green build to
> merge.

The reason I'm interested in this change is because this project
already appears to adopt this policy of bottom-up flakiness reduction,
and I don't think it has been effective enough to be able to enforce
requiring a green build.
I think that it is better to improve the incentives for flakiness
reduction, wait for flakiness to improve, and then later enforce a
green build to merge. In that context, the partial builds are a
temporary change to help us get to the desired end-goal.

Thanks,
Greg

On Tue, Jun 6, 2023 at 2:51 AM David Jacot  wrote:
>
> Hey Greg,
>
> Thanks for bringing this up.
>
> I am not sure to understand the benefit of the parallele trigger of a
> subset of the tests. Indeed, they will be more likely to pass but the
> downside is that folks may start to only rely on that signal and commit
> without looking at the full test suite. This seems dangerous to me.
>
> However, I agree that we have an issue with our builds. We have way too
> many flaky tests. I would rather focus on trying to address this first. If
> we can stabilize them, I wonder if we should also enforce a green build to
> merge.
>
> Best,
> David
>
>
>
> On Mon, Jun 5, 2023 at 7:47 PM Greg Harris 
> wrote:
>
> > Hey all,
> >
> > I've been working on test flakiness recently, and I've been trying to
> > come up with ways to tackle the issue top-down as well as bottom-up,
> > and I'm interested to hear your thoughts on an idea.
> >
> > In addition to the current full-suite runs, can we in parallel trigger
> > a smaller test run which has only a relevant subset of tests? For
> > example, if someone is working on one sub-module, the CI would only
> > run tests in that module.
> >
> > I think this would be more likely to pass than the full suite due to
> > the fewer tests failing probabilistically, and would improve the
> > signal-to-noise ratio of the summary pass/fail marker on GitHub. This
> > should also be shorter to execute than the full suite, allowing for
> > faster cycle-time than the current full suite encourages.
> >
> > This would also strengthen the incentive for contributors specializing
> > in a module to de-flake tests, as they are rewarded with a tangible
> > improvement within their area of the project. Currently, even the
> > modules with the most reliable tests receive consistent CI failures
> > from other less reliable modules.
> >
> > I believe this is possible, even if there isn't an off-the-shelf
> > solution for it. We can learn of the changed files via a git diff, map
> > that to modules containing those files, and then execute the tests
> > just for those modules with gradle. GitHub also permits showing
> > multiple "checks" so that we can emit both the full-suite and partial
> > test results.
> >
> > Thanks,
> > Greg
> >

Re: [VOTE] 3.5.0 RC1

[Federico Valeri]

Hi Mickael, I did the following checks:

- Signature, checksum, licenses
- Build from source with Java 17 and Scala 2.13
- Full unit and integration test suites
- Java app with staging Maven artifacts and multi-node cluster

LGTM, apart from the already mentioned "classgraph" license that can be removed.

+1 (non binding)

Thanks
Fede

On Tue, Jun 6, 2023 at 9:39 AM Luke Chen  wrote:
>
> Hi Mickael,
>
> I ran the following validation steps:
> - Built from source with Java 17 and Scala 2.13
> - Signatures and hashes of the artifacts generated
> - Navigated through Javadoc including links to JDK classes
> - Run the quickstart in KRaft and Zookeeper mode
>
> +1 (binding) from me.
> Thanks for running the release.
>
> Luke
>
> On Tue, Jun 6, 2023 at 4:23 AM Josep Prat 
> wrote:
>
> > Hi Mickael,
> >
> > I ran the following validation steps:
> > - Built from source with Java 11 and Scala 2.13
> > - Signatures and hashes of the artifacts generated
> > - Navigated through Javadoc including links to JDK classes
> > - Run the unit tests
> > - Run integration tests
> > - Run the quickstart in KRaft and Zookeeper mode
> > -- For KRaft, I looked at the process running Kafka and confirmed that the
> > spamming log message is not present anymore ("Generated a metadata delta
> > between...")
> > - Checked the contents of LICENSE-binary against the lib folder
> > -- I found that the LICENSE-binary has a reference to classgraph-4.8.138
> > but I fail to find this library in the lib folder. Trying to backtrack when
> > it was added, it seems it was done here:
> >
> > https://github.com/apache/kafka/commit/3a2ac267178c5464e41b94fcbb2dd897212812bd
> > but I fail to find this library in the 3.3.1 lib folder (3.3.0 was a dead
> > release).
> >
> > I'm not sure this qualifies as a blocker for the release though, I leave it
> > up to you.
> >
> > Best,
> >
> > On Mon, Jun 5, 2023 at 3:39 PM Mickael Maison  wrote:
> >
> > > Hello Kafka users, developers and client-developers,
> > >
> > > This is the second candidate for release of Apache Kafka 3.5.0. Some
> > > of the major features include:
> > > - KIP-710: Full support for distributed mode in dedicated MirrorMaker
> > > 2.0 clusters
> > > - KIP-881: Rack-aware Partition Assignment for Kafka Consumers
> > > - KIP-887: Add ConfigProvider to make use of environment variables
> > > - KIP-889: Versioned State Stores
> > > - KIP-894: Use incrementalAlterConfig for syncing topic configurations
> > > - KIP-900: KRaft kafka-storage.sh API additions to support SCRAM for
> > > Kafka Brokers
> > >
> > > Release notes for the 3.5.0 release:
> > > https://home.apache.org/~mimaison/kafka-3.5.0-rc1/RELEASE_NOTES.html
> > >
> > > *** Please download, test and vote by Friday June 9, 5pm PT
> > >
> > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > https://kafka.apache.org/KEYS
> > >
> > > * Release artifacts to be voted upon (source and binary):
> > > https://home.apache.org/~mimaison/kafka-3.5.0-rc1/
> > >
> > > * Maven artifacts to be voted upon:
> > > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > >
> > > * Javadoc:
> > > https://home.apache.org/~mimaison/kafka-3.5.0-rc1/javadoc/
> > >
> > > * Tag to be voted upon (off 3.5 branch) is the 3.5.0 tag:
> > > https://github.com/apache/kafka/releases/tag/3.5.0-rc1
> > >
> > > * Documentation:
> > > https://kafka.apache.org/35/documentation.html
> > >
> > > * Protocol:
> > > https://kafka.apache.org/35/protocol.html
> > >
> > > * Successful Jenkins builds for the 3.5 branch:
> > > Unit/integration tests: I'm struggling to get all tests to pass in the
> > > same build. I'll run a few more builds to ensure each test pass at
> > > least once in the CI. All tests passed locally.
> > > System tests: The build is still running, I'll send an update once I
> > > have the results.
> > >
> > > Thanks,
> > > Mickael
> > >
> >
> >
> > --
> > [image: Aiven] 
> >
> > *Josep Prat*
> > Open Source Engineering Director, *Aiven*
> > josep.p...@aiven.io   |   +491715557497
> > aiven.io    |    > >
> >      <
> > https://twitter.com/aiven_io>
> > *Aiven Deutschland GmbH*
> > Alexanderufer 3-7, 10117 Berlin
> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > Amtsgericht Charlottenburg, HRB 209739 B
> >

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #1902

[Apache Jenkins Server]

See 

Re: [VOTE] KIP-923: Add A Grace Period to Stream Table Join

[Walker Carlson]

Hi all,

This vote thread has been open over 72 hours, and has sufficient votes so
I'll close the voting at this time.

+4 binding votes
+2 non-binding votes

KIP-923 has PASSED.

Thanks all for your votes
Walker

On Tue, Jun 6, 2023 at 8:13 AM John Roesler  wrote:

> Thanks for the KIP, Walker!
>
> I’m +1 (binding)
>
> Thanks,
> John
>
> On Mon, Jun 5, 2023, at 13:39, Victoria Xia wrote:
> > Hi Walker,
> >
> > Thanks for the KIP! Left a clarification question on the discussion
> thread
> > just now but it's about an implementation detail, so I don't think it
> > changes anything in this vote thread.
> >
> > +1 (non-binding)
> >
> > Cheers,
> > Victoria
> >
> > On Mon, Jun 5, 2023 at 10:23 AM Bill Bejeck  wrote:
> >
> >> Hi Walker,
> >>
> >> Thanks for the KIP.
> >>
> >> I've caught up on the discussion thread and I'm satisfied with all
> >> responses.
> >>
> >> +1(binding)
> >>
> >> -Bill
> >>
> >> On Mon, Jun 5, 2023 at 10:20 AM Bruno Cadonna 
> wrote:
> >>
> >> > Hi Walker,
> >> >
> >> > Thank you for the KIP!
> >> >
> >> > +1 (binding)
> >> >
> >> > Best,
> >> > Bruno
> >> >
> >> > On 24.05.23 23:00, Walker Carlson wrote:
> >> > > Hello everybody,
> >> > >
> >> > > I'm opening the vote on KIP-923 here
> >> > > .
> >> > >
> >> > > If we have more to discus please continue the discussion on the
> >> existing
> >> > > thread
> >> https://www.mail-archive.com/dev@kafka.apache.org/msg130657.html
> >> > >
> >> > > best,
> >> > > Walker
> >> > >
> >> >
> >>
>

Re: [DISCUSS] KIP-923: Add A Grace Period to Stream Table Join

[Walker Carlson]

Good Point Victoria. I just removed the compacted topic mention from the
KIP. I agree with Burno about using a normal topic and deleting records
that have been processed.

On Tue, Jun 6, 2023 at 2:28 AM Bruno Cadonna  wrote:

> Hi,
>
> another idea that came to my mind. Instead of using a compacted topic,
> the buffer could use a non-compacted topic and regularly delete records
> before a given offset as Streams does for repartition topics.
>
> Best,
> Bruno
>
> On 05.06.23 21:48, Bruno Cadonna wrote:
> > Hi Victoria,
> >
> > that is a good point!
> >
> > I think, the topic needs to be a compacted topic to be able to get rid
> > of records that are evicted from the buffer. So the key might be
> > something with the key, the timestamp, and a sequence number to
> > distinguish between records with the same key and same timestamp.
> >
> > Just an idea! Maybe Walker comes up with something better.
> >
> > Best,
> > Bruno
> >
> > On 05.06.23 20:38, Victoria Xia wrote:
> >> Hi Walker,
> >>
> >> Thanks for the latest updates! The KIP looks great. Just one question
> >> about
> >> the changelog topic for the join buffer: The KIP says "When a failure
> >> occurs the buffer will try to recover from an OffsetCheckpoint if
> >> possible.
> >> If not it will reload the buffer from a compacted change-log topic."
> This
> >> is a new changelog topic that will be introduced specifically for the
> >> join
> >> buffer, right? Why is the changelog topic compacted? What are the keys?
> I
> >> am confused because the buffer contains records from the stream-side
> >> of the
> >> join, for which multiple records with the same key should be treated as
> >> separate updates will all must be tracked in the buffer, rather than
> >> updates which replace each other.
> >>
> >> Thanks,
> >> Victoria
> >>
> >> On Mon, Jun 5, 2023 at 1:47 AM Bruno Cadonna 
> wrote:
> >>
> >>> Hi Walker,
> >>>
> >>> Thanks once more for the updates to the KIP!
> >>>
> >>> Do you also plan to expose metrics for the buffer?
> >>>
> >>> Best,
> >>> Bruno
> >>>
> >>> On 02.06.23 17:16, Walker Carlson wrote:
>  Hello Bruno,
> 
>  I think this covers your questions. Let me know what you think
> 
>  2.
>  We can use a changelog topic. I think we can treat it like any other
> >>> store
>  and recover in the usual manner. Also implementation is on disk
> 
>  3.
>  The description is in the public interfaces description. I will copy
> it
>  into the proposed changes as well.
> 
>  This is a bit of an implementation detail that I didn't want to add
>  into
>  the kip, but the record will be added to the buffer to keep the stream
> >>> time
>  consistent, it will just be ejected immediately. If of course if this
>  causes performance issues we will skip this step and track stream time
>  separately. I will update the kip to say that stream time advances
>  when a
>  stream record enters the node.
> 
>  Also, yes, updated.
> 
>  5.
>  No there is no difference right now, everything gets processed as it
> >>> comes
>  in and tries to find a record for its time stamp.
> 
>  Walker
> 
>  On Fri, Jun 2, 2023 at 6:41 AM Bruno Cadonna 
>  wrote:
> 
> > Hi Walker,
> >
> > Thanks for the updates!
> >
> > 2.
> > It is still not clear to me how a failure is handled. I do not
> > understand what you mean by "recover from an OffsetCheckpoint".
> >
> > My understanding is that the buffer needs to be replicated into its
> > own
> > Kafka topic. The input topic is not enough. The offset of a record is
> > added to the offsets to commit once the record is streamed through
> the
> > subtopology. That means once the record is added to the buffer its
> > offset is added to the offsets to commit -- independently of
> > whether the
> > record was evicted from the buffer and sent to the join node or not.
> > Now, let's assume the following scenario
> > 1. a record is read from the input topic and added to the buffer, but
> > not evicted to be processed by the join node.
> > 2. When the processing of the subtopology finishes the offset of the
> > record is added to the offsets to commit.
> > 3. A commit happens.
> > 4. A failure happens
> >
> > After the failure the buffer is empty but the record will not be read
> > anymore from the input topic since its offset has been already
> > committed. The record is lost.
> > One solution to avoid the loss is to recreate the buffer from a
> > compacted Kafka topic as we do for suppression buffers. I do not
> > think,
> > we need any offset checkpoint here since, we keep the buffer in
> > memory,
> > right? Or do you plan to back the buffer with a persistent store?
> Even
> > in that case, a compacted Kafka topic would be needed.
> >
> >
> > 3.
> >From the KIP it is 

Re: [DISCUSS] Regarding Old PRs

[Chris Egerton]

Hi Josep,

Thanks for bringing this up! Will try to keep things brief.

I'm generally in favor of this initiative. A couple ideas that I really
liked: requiring a component label (producer, consumer, connect, streams,
etc.) before closing, and disabling auto-close (i.e., automatically tagging
PRs as stale, but leaving it to a human being to actually close them).

We might replace the "stale" label with a "close-by-" label so that
it becomes even easier for us to find the PRs that are ready to be closed
(as opposed to the ones that have just been labeled as stale without giving
the contributor enough time to respond).

I've also gone ahead and closed some of my stale PRs. Others I've
downgraded to draft to signal that I'd like to continue to pursue them, but
have to iron out merge conflicts first. For the last ones, I'll ping for
review.

One question that came to mind--do we want to distinguish between regular
and draft PRs? I'm guessing not, since they still add up to the total PR
count against the project, but since they do also implicitly signal that
they're not intended for review (yet) it may be friendlier to leave them up.

Cheers,

Chris

On Tue, Jun 6, 2023 at 10:18 AM Mickael Maison 
wrote:

> Hi Josep,
>
> Thanks for looking into this. This is clearly one aspect where we need
> to improve.
>
> We had a similar initiative last year
> (https://lists.apache.org/thread/66yj9m6tcyz8zqb3lqlbnr386bqwsopt) and
> we closed many PRs. Unfortunately we did not follow up with a process
> or automation and we are back to the same situation.
>
> Manually reviewing all these PRs is a huge task, so I think we should
> at least partially automate it. I'm not sure if we should manually
> review the oldest PRs (pre 2020). There's surely many interesting
> things but I wonder if we should instead focus on the more recent ones
> as they have a higher chance of 1) still making sense, 2) getting
> updates from their authors, 3) needing less rebasing. If something has
> been broken since 2016 but we never bothered to fix the PR it means it
> can't be anything critical!
>
> Finally as Colin mentioned, it looks like a non negligible chunk of
> stale PRs comes from committers and regular contributors. So I'd
> suggest we each try to clean our own backlog too.
>
> I wonder if we also need to do something in JIRA. Querying for
> unresolved tickets returns over 4000 items. Considering we're not
> quite at KAFKA-15000 yet, that's a lot.
>
> Thanks,
> Mickael
>
>
> On Tue, Jun 6, 2023 at 11:35 AM Josep Prat 
> wrote:
> >
> > Hi Devs,
> > I would say we can split the problem in 2.
> >
> > Waiting for Author feedback:
> > We could have a bot that would ping authors for the cases where we have
> PRs
> > that are stalled and have either:
> > - Merge conflict
> > - Unaddressed reviews
> >
> > Waiting for reviewers:
> > For the PRs where we have no reviewers and there are no conflicts, I
> think
> > we would need some human interaction to determine modules (maybe this can
> > be automated) and ping people who could review.
> >
> > What do you think?
> >
> > Best,
> >
> > On Tue, Jun 6, 2023 at 11:30 AM Josep Prat  wrote:
> >
> > > Hi Nikolay,
> > >
> > > With a bot it will be complicated to determine what to do when the PR
> > > author is waiting for a reviewer. If a person goes over them, can
> check if
> > > they are waiting for reviews and tag the PR accordingly and maybe ping
> a
> > > maintainer.
> > > If you look at my last email I described a flow (but AFAIU it will work
> > > only if a human executes it) where the situation you point out would be
> > > covered.
> > >
> > > ———
> > > Josep Prat
> > >
> > > Aiven Deutschland GmbH
> > >
> > > Alexanderufer 3-7, 10117 Berlin
> > >
> > > Amtsgericht Charlottenburg, HRB 209739 B
> > >
> > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > >
> > > m: +491715557497
> > >
> > > w: aiven.io
> > >
> > > e: josep.p...@aiven.io
> > >
> > > On Tue, Jun 6, 2023, 11:20 Николай Ижиков  wrote:
> > >
> > >> Hello.
> > >>
> > >> What is actions of contributor if no feedback given? [1], [2]
> > >>
> > >> [1] https://github.com/apache/kafka/pull/13278
> > >> [2] https://github.com/apache/kafka/pull/13247
> > >>
> > >> > 2 июня 2023 г., в 23:38, David Arthur 
> написал(а):
> > >> >
> > >> > I think this is a great idea. If we don’t want the auto-close
> > >> > functionality, we can set it to -1
> > >> >
> > >> > I realize this isn’t a vote, but I’m +1 on this
> > >> >
> > >> > -David
> > >> >
> > >> > On Fri, Jun 2, 2023 at 15:34 Colin McCabe 
> wrote:
> > >> >
> > >> >> That should read "30 days without activity"
> > >> >>
> > >> >> (I am assuming we have the ability to determine when a PR was last
> > >> updated
> > >> >> on GH)
> > >> >>
> > >> >> best,
> > >> >> Colin
> > >> >>
> > >> >> On Fri, Jun 2, 2023, at 12:32, Colin McCabe wrote:
> > >> >>> Hi all,
> > >> >>>
> > >> >>> Looking at GitHub, I have a bunch of Kafka PRs of my own that I've
> > >> >>> allowed to become stale, and I 

Re: [DISCUSS] KIP-936: Throttle number of active PIDs

[Haruki Okada]

Hi, Omnia.

Thanks for the KIP.
The feature sounds indeed helpful and the strategy to use bloom-filter
looks good.

I have three questions:

1. Do you have any memory-footprint estimation
for TimeControlledBloomFilter?
* If I read the KIP correctly, TimeControlledBloomFilter will be
allocated per KafkaPrincipal so the size should be reasonably small
considering clusters which have a large number of users.
* i.e. What false-positive rate do you plan to choose as the default?
2. What do you think about rotating windows on produce-requests arrival
instead of scheduler?
* If we do rotation in scheduler threads, my concern is potential
scheduler threads occupation which could make other background tasks to
delay
3. Why the default producer.id.quota.window.size.seconds is 1 hour?
* Unlike other quota types (1 second)

Thanks,

2023年6月6日(火) 23:55 Omnia Ibrahim :

> Hi everyone,
> I want to start the discussion of the KIP-936 to throttle the number of
> active PIDs per KafkaPrincipal. The proposal is here
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-936%3A+Throttle+number+of+active+PIDs
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-936%3A+Throttle+number+of+active+PIDs
> >
>
> Thanks for your time and feedback.
> Omnia
>


-- 

Okada Haruki
ocadar...@gmail.com

Re: [DISCUSS] Adding non-committers as Github collaborators

[Divij Vaidya]

Hey John

What does the invite do? I did not receive any invite either but I have
been able to add labels, close PRs etc. since I was added to the
contributors list (perhaps because I already have an apache LDAP account?).
I am working with the assumption that things are working for me even if I
didn't have to accept an invite. Is that a correct assumption?

--
Divij Vaidya



On Tue, Jun 6, 2023 at 5:32 PM John Roesler  wrote:

> Hello all,
>
> I put in a ticket with Apache Infra to re-send these invites, and they
> told me I should just remove the usernames in one commit and then re-add
> them in a subsequent commit to trigger the invites again.
>
> I will go ahead and do this for the users who requested it on this thread
> (Greg and Andrew, as well as for Victoria who asked me about it
> separately). If there is anyone else who needs a re-send, please let us
> know.
>
> I'm sorry for the confusion, Hao. The docs claimed we could add 20 users,
> but when I actually checked in the file, I got an automated notification
> that we could actually only have 10.
>
> As for triggering the build, I don't believe you'll be able to log in to
> Jenkins, but you should be able to say "retest this please" in a PR comment
> to trigger it. Apparently, that doesn't work anymore, though. I'll file an
> Infra ticket for it.
>
> Thanks all,
> John
>
> On Fri, Jun 2, 2023, at 18:46, Hao Li wrote:
> > Hi Luke,
> >
> > Sorry for the late reply. Can you also add me to the whitelist? I believe
> > I'm supposed to be there as well. Matthias and John can vouch for me :)
> My
> > github ID is "lihaosky".
> >
> > Thanks,
> > Hao
> >
> > On Fri, Jun 2, 2023 at 4:43 PM Greg Harris  >
> > wrote:
> >
> >> Luke,
> >>
> >> I see that the PR has been merged, but I don't believe it re-sent the
> >> invitation.
> >>
> >> Thanks
> >> Greg
> >>
> >>
> >> On Wed, May 31, 2023 at 6:46 PM Luke Chen  wrote:
> >> >
> >> > Hi Greg and Andrew,
> >> >
> >> > Sorry, I don't know how to re-sent the invitation.
> >> > It looks like it is auto sent after the .asf.yaml updated.
> >> > Since updating collaborator list is part of release process based on
> the
> >> doc
> >> > , I just created a new list
> and
> >> > opened a PR:
> >> > https://github.com/apache/kafka/pull/13790
> >> >
> >> > Hope that after this PR merged, you'll get a new invitation.
> >> >
> >> > Thanks.
> >> > Luke
> >> >
> >> > On Thu, Jun 1, 2023 at 5:27 AM Andrew Grant
>  >> >
> >> > wrote:
> >> >
> >> > > Hi all,
> >> > > Like Greg I also received an invitation to collaborate but was too
> >> slow to
> >> > > accept the invite :( I'm also wondering if there's a way to resend
> the
> >> > > invite? I'm andymg3 on GitHub.
> >> > > Thanks,
> >> > > Andrew
> >> > >
> >> > > On Tue, May 30, 2023 at 12:12 PM Greg Harris
> >>  >> > > >
> >> > > wrote:
> >> > >
> >> > > > Hey all,
> >> > > >
> >> > > > I received an invitation to collaborate on apache/kafka, but let
> the
> >> > > > invitation expire after 7 days.
> >> > > > Is there a workflow for refreshing the invite, or is an admin
> able to
> >> > > > manually re-invite me?
> >> > > > I'm gharris1727 on github.
> >> > > >
> >> > > > Thanks!
> >> > > > Greg
> >> > > >
> >> > > > On Wed, May 24, 2023 at 9:32 AM Justine Olshan
> >> > > >  wrote:
> >> > > > >
> >> > > > > Hey Yash,
> >> > > > > I'm not sure how it used to be for sure, but I do remember we
> used
> >> to
> >> > > > have
> >> > > > > a different build system. I wonder if this used to work with the
> >> old
> >> > > > build
> >> > > > > system and not any more.
> >> > > > > I'd be curious if other projects have something similar and how
> it
> >> > > works.
> >> > > > >
> >> > > > > Thanks,
> >> > > > > Justine
> >> > > > >
> >> > > > > On Wed, May 24, 2023 at 9:22 AM Yash Mayya <
> yash.ma...@gmail.com>
> >> > > wrote:
> >> > > > >
> >> > > > > > Hi Justine,
> >> > > > > >
> >> > > > > > Thanks for the response. Non-committers don't have Apache
> >> accounts;
> >> > > > are you
> >> > > > > > suggesting that there wasn't a need to sign in earlier and a
> >> change
> >> > > in
> >> > > > this
> >> > > > > > policy is restricting collaborators from triggering Jenkins
> >> builds?
> >> > > > > >
> >> > > > > > Thanks,
> >> > > > > > Yash
> >> > > > > >
> >> > > > > > On Wed, May 24, 2023 at 9:30 PM Justine Olshan
> >> > > > > > 
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > > > Yash,
> >> > > > > > >
> >> > > > > > > When I rebuild, I go to the CloudBees CI page and I have to
> >> log in
> >> > > > with
> >> > > > > > my
> >> > > > > > > apache account.
> >> > > > > > > Not sure if the change in the build system or the need to
> sign
> >> in
> >> > > is
> >> > > > part
> >> > > > > > > of the problem.
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > On Wed, May 24, 2023 at 4:54 AM Federico Valeri <
> >> > > > fedeval...@gmail.com>
> >> > > > > > > wrote:
> >> > > > > > >
> >> > > > > > > > +1 on Divij suggestions
> >> > > > > > > >
> 

[jira] [Reopened] (KAFKA-15021) KRaft controller increases leader epoch when shrinking ISR

[Jira]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

José Armando García Sancio reopened KAFKA-15021:


> KRaft controller increases leader epoch when shrinking ISR
> --
>
> Key: KAFKA-15021
> URL: https://issues.apache.org/jira/browse/KAFKA-15021
> Project: Kafka
>  Issue Type: Bug
>  Components: controller, kraft
>Reporter: José Armando García Sancio
>Assignee: José Armando García Sancio
>Priority: Major
>
> When the KRaft controller shrinks the ISR it also forces the leader epoch to 
> increase. This is unnecessary and cases all of the follower replica fetches 
> to get invalidated.
> Here is an example trace of this behavior after replica 8 was shutdown:
> {code:java}
> kafka-dump-log --cluster-metadata-decoder --files 
> __cluster_metadata-0/38589501.log | grep Pd7wMb4lSkKI00--SrWNXw
> ...
> | offset: 38655592 CreateTime: 1683849857362 keySize: -1 valueSize: 41 
> sequence: -1 headerKeys: [] payload: 
> {"type":"PARTITION_CHANGE_RECORD","version":0,"data":{"partitionId":7,"topicId":"Pd7wMb4lSkKI00--SrWNXw","isr":[3,1],"leader":1}}
> | offset: 38655593 CreateTime: 1683849857362 keySize: -1 valueSize: 41 
> sequence: -1 headerKeys: [] payload: 
> {"type":"PARTITION_CHANGE_RECORD","version":0,"data":{"partitionId":5,"topicId":"Pd7wMb4lSkKI00--SrWNXw","isr":[0,4],"leader":4}}
> | offset: 38655594 CreateTime: 1683849857362 keySize: -1 valueSize: 41 
> sequence: -1 headerKeys: [] payload: 
> {"type":"PARTITION_CHANGE_RECORD","version":0,"data":{"partitionId":6,"topicId":"Pd7wMb4lSkKI00--SrWNXw","isr":[0,1],"leader":0}}
> | offset: 38656159 CreateTime: 1683849974945 keySize: -1 valueSize: 39 
> sequence: -1 headerKeys: [] payload: 
> {"type":"PARTITION_CHANGE_RECORD","version":0,"data":{"partitionId":7,"topicId":"Pd7wMb4lSkKI00--SrWNXw","isr":[3,1,8]}}
> | offset: 38656256 CreateTime: 1683849994297 keySize: -1 valueSize: 39 
> sequence: -1 headerKeys: [] payload: 
> {"type":"PARTITION_CHANGE_RECORD","version":0,"data":{"partitionId":5,"topicId":"Pd7wMb4lSkKI00--SrWNXw","isr":[0,4,8]}}
> | offset: 38656299 CreateTime: 1683849997139 keySize: -1 valueSize: 39 
> sequence: -1 headerKeys: [] payload: 
> {"type":"PARTITION_CHANGE_RECORD","version":0,"data":{"partitionId":6,"topicId":"Pd7wMb4lSkKI00--SrWNXw","isr":[0,1,8]}}
> | offset: 38657003 CreateTime: 1683850157379 keySize: -1 valueSize: 30 
> sequence: -1 headerKeys: [] payload: 
> {"type":"PARTITION_CHANGE_RECORD","version":0,"data":{"partitionId":7,"topicId":"Pd7wMb4lSkKI00--SrWNXw","leader":8}}
>  {code}
> Also, notice how the leader epoch was not increased when the ISR was expanded.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15021) KRaft controller increases leader epoch when shrinking ISR

[Jira]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

José Armando García Sancio resolved KAFKA-15021.

Resolution: Fixed

> KRaft controller increases leader epoch when shrinking ISR
> --
>
> Key: KAFKA-15021
> URL: https://issues.apache.org/jira/browse/KAFKA-15021
> Project: Kafka
>  Issue Type: Bug
>  Components: controller, kraft
>Reporter: José Armando García Sancio
>Assignee: José Armando García Sancio
>Priority: Major
>
> When the KRaft controller shrinks the ISR it also forces the leader epoch to 
> increase. This is unnecessary and cases all of the follower replica fetches 
> to get invalidated.
> Here is an example trace of this behavior after replica 8 was shutdown:
> {code:java}
> kafka-dump-log --cluster-metadata-decoder --files 
> __cluster_metadata-0/38589501.log | grep Pd7wMb4lSkKI00--SrWNXw
> ...
> | offset: 38655592 CreateTime: 1683849857362 keySize: -1 valueSize: 41 
> sequence: -1 headerKeys: [] payload: 
> {"type":"PARTITION_CHANGE_RECORD","version":0,"data":{"partitionId":7,"topicId":"Pd7wMb4lSkKI00--SrWNXw","isr":[3,1],"leader":1}}
> | offset: 38655593 CreateTime: 1683849857362 keySize: -1 valueSize: 41 
> sequence: -1 headerKeys: [] payload: 
> {"type":"PARTITION_CHANGE_RECORD","version":0,"data":{"partitionId":5,"topicId":"Pd7wMb4lSkKI00--SrWNXw","isr":[0,4],"leader":4}}
> | offset: 38655594 CreateTime: 1683849857362 keySize: -1 valueSize: 41 
> sequence: -1 headerKeys: [] payload: 
> {"type":"PARTITION_CHANGE_RECORD","version":0,"data":{"partitionId":6,"topicId":"Pd7wMb4lSkKI00--SrWNXw","isr":[0,1],"leader":0}}
> | offset: 38656159 CreateTime: 1683849974945 keySize: -1 valueSize: 39 
> sequence: -1 headerKeys: [] payload: 
> {"type":"PARTITION_CHANGE_RECORD","version":0,"data":{"partitionId":7,"topicId":"Pd7wMb4lSkKI00--SrWNXw","isr":[3,1,8]}}
> | offset: 38656256 CreateTime: 1683849994297 keySize: -1 valueSize: 39 
> sequence: -1 headerKeys: [] payload: 
> {"type":"PARTITION_CHANGE_RECORD","version":0,"data":{"partitionId":5,"topicId":"Pd7wMb4lSkKI00--SrWNXw","isr":[0,4,8]}}
> | offset: 38656299 CreateTime: 1683849997139 keySize: -1 valueSize: 39 
> sequence: -1 headerKeys: [] payload: 
> {"type":"PARTITION_CHANGE_RECORD","version":0,"data":{"partitionId":6,"topicId":"Pd7wMb4lSkKI00--SrWNXw","isr":[0,1,8]}}
> | offset: 38657003 CreateTime: 1683850157379 keySize: -1 valueSize: 30 
> sequence: -1 headerKeys: [] payload: 
> {"type":"PARTITION_CHANGE_RECORD","version":0,"data":{"partitionId":7,"topicId":"Pd7wMb4lSkKI00--SrWNXw","leader":8}}
>  {code}
> Also, notice how the leader epoch was not increased when the ISR was expanded.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Adding non-committers as Github collaborators

[John Roesler]

Hello all,

I put in a ticket with Apache Infra to re-send these invites, and they told me 
I should just remove the usernames in one commit and then re-add them in a 
subsequent commit to trigger the invites again.

I will go ahead and do this for the users who requested it on this thread (Greg 
and Andrew, as well as for Victoria who asked me about it separately). If there 
is anyone else who needs a re-send, please let us know.

I'm sorry for the confusion, Hao. The docs claimed we could add 20 users, but 
when I actually checked in the file, I got an automated notification that we 
could actually only have 10.

As for triggering the build, I don't believe you'll be able to log in to 
Jenkins, but you should be able to say "retest this please" in a PR comment to 
trigger it. Apparently, that doesn't work anymore, though. I'll file an Infra 
ticket for it.

Thanks all,
John

On Fri, Jun 2, 2023, at 18:46, Hao Li wrote:
> Hi Luke,
>
> Sorry for the late reply. Can you also add me to the whitelist? I believe
> I'm supposed to be there as well. Matthias and John can vouch for me :) My
> github ID is "lihaosky".
>
> Thanks,
> Hao
>
> On Fri, Jun 2, 2023 at 4:43 PM Greg Harris 
> wrote:
>
>> Luke,
>>
>> I see that the PR has been merged, but I don't believe it re-sent the
>> invitation.
>>
>> Thanks
>> Greg
>>
>>
>> On Wed, May 31, 2023 at 6:46 PM Luke Chen  wrote:
>> >
>> > Hi Greg and Andrew,
>> >
>> > Sorry, I don't know how to re-sent the invitation.
>> > It looks like it is auto sent after the .asf.yaml updated.
>> > Since updating collaborator list is part of release process based on the
>> doc
>> > , I just created a new list and
>> > opened a PR:
>> > https://github.com/apache/kafka/pull/13790
>> >
>> > Hope that after this PR merged, you'll get a new invitation.
>> >
>> > Thanks.
>> > Luke
>> >
>> > On Thu, Jun 1, 2023 at 5:27 AM Andrew Grant > >
>> > wrote:
>> >
>> > > Hi all,
>> > > Like Greg I also received an invitation to collaborate but was too
>> slow to
>> > > accept the invite :( I'm also wondering if there's a way to resend the
>> > > invite? I'm andymg3 on GitHub.
>> > > Thanks,
>> > > Andrew
>> > >
>> > > On Tue, May 30, 2023 at 12:12 PM Greg Harris
>> > > > >
>> > > wrote:
>> > >
>> > > > Hey all,
>> > > >
>> > > > I received an invitation to collaborate on apache/kafka, but let the
>> > > > invitation expire after 7 days.
>> > > > Is there a workflow for refreshing the invite, or is an admin able to
>> > > > manually re-invite me?
>> > > > I'm gharris1727 on github.
>> > > >
>> > > > Thanks!
>> > > > Greg
>> > > >
>> > > > On Wed, May 24, 2023 at 9:32 AM Justine Olshan
>> > > >  wrote:
>> > > > >
>> > > > > Hey Yash,
>> > > > > I'm not sure how it used to be for sure, but I do remember we used
>> to
>> > > > have
>> > > > > a different build system. I wonder if this used to work with the
>> old
>> > > > build
>> > > > > system and not any more.
>> > > > > I'd be curious if other projects have something similar and how it
>> > > works.
>> > > > >
>> > > > > Thanks,
>> > > > > Justine
>> > > > >
>> > > > > On Wed, May 24, 2023 at 9:22 AM Yash Mayya 
>> > > wrote:
>> > > > >
>> > > > > > Hi Justine,
>> > > > > >
>> > > > > > Thanks for the response. Non-committers don't have Apache
>> accounts;
>> > > > are you
>> > > > > > suggesting that there wasn't a need to sign in earlier and a
>> change
>> > > in
>> > > > this
>> > > > > > policy is restricting collaborators from triggering Jenkins
>> builds?
>> > > > > >
>> > > > > > Thanks,
>> > > > > > Yash
>> > > > > >
>> > > > > > On Wed, May 24, 2023 at 9:30 PM Justine Olshan
>> > > > > > 
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Yash,
>> > > > > > >
>> > > > > > > When I rebuild, I go to the CloudBees CI page and I have to
>> log in
>> > > > with
>> > > > > > my
>> > > > > > > apache account.
>> > > > > > > Not sure if the change in the build system or the need to sign
>> in
>> > > is
>> > > > part
>> > > > > > > of the problem.
>> > > > > > >
>> > > > > > >
>> > > > > > > On Wed, May 24, 2023 at 4:54 AM Federico Valeri <
>> > > > fedeval...@gmail.com>
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > > > +1 on Divij suggestions
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On Wed, May 24, 2023 at 12:04 PM Divij Vaidya <
>> > > > divijvaidy...@gmail.com
>> > > > > > >
>> > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > Hey folks
>> > > > > > > > >
>> > > > > > > > > A week into this experiment, I am finding the ability to
>> add
>> > > > labels,
>> > > > > > > > > request for reviewers and ability to close PRs very useful.
>> > > > > > > > >
>> > > > > > > > > 1. May I suggest an improvement to the process by
>> requesting
>> > > for
>> > > > some
>> > > > > > > > > guidance on the interest areas for various committers. This
>> > > would
>> > > > > > help
>> > > > > > > us
>> > > > > > > > > request for reviews from the right set of individuals.
>> > > > > > > > > As a reference, 

[jira] [Resolved] (KAFKA-14791) Create a builder class for PartitionRegistration

[Andrew Grant (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Grant resolved KAFKA-14791.
--
Resolution: Fixed

> Create a builder class for PartitionRegistration
> 
>
> Key: KAFKA-14791
> URL: https://issues.apache.org/jira/browse/KAFKA-14791
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Andrew Grant
>Assignee: Andrew Grant
>Priority: Minor
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1901

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 481169 lines...]
[2023-06-06T14:59:51.388Z] [WARNING] Plugin validation issues were detected in 
2 plugin(s)
[2023-06-06T14:59:51.388Z] [WARNING] 
[2023-06-06T14:59:51.388Z] [WARNING]  * 
org.apache.maven.plugins:maven-compiler-plugin:3.1
[2023-06-06T14:59:51.388Z] [WARNING]  * 
org.apache.maven.plugins:maven-resources-plugin:3.3.0
[2023-06-06T14:59:51.388Z] [WARNING] 
[2023-06-06T14:59:51.388Z] [WARNING] For more or less details, use 
'maven.plugin.validation' property with one of the values (case insensitive): 
[BRIEF, DEFAULT, VERBOSE]
[2023-06-06T14:59:51.388Z] [WARNING] 
[2023-06-06T14:59:51.895Z] > Task :streams:testSrcJar
[2023-06-06T14:59:51.896Z] > Task 
:streams:publishMavenJavaPublicationToMavenLocal
[2023-06-06T14:59:51.896Z] > Task :streams:publishToMavenLocal
[2023-06-06T14:59:51.896Z] 
[2023-06-06T14:59:51.896Z] Deprecated Gradle features were used in this build, 
making it incompatible with Gradle 9.0.
[2023-06-06T14:59:51.896Z] 
[2023-06-06T14:59:51.896Z] You can use '--warning-mode all' to show the 
individual deprecation warnings and determine if they come from your own 
scripts or plugins.
[2023-06-06T14:59:51.896Z] 
[2023-06-06T14:59:51.896Z] See 
https://docs.gradle.org/8.1.1/userguide/command_line_interface.html#sec:command_line_warnings
[2023-06-06T14:59:51.896Z] 
[2023-06-06T14:59:51.896Z] BUILD SUCCESSFUL in 5m 10s
[2023-06-06T14:59:51.896Z] 89 actionable tasks: 35 executed, 54 up-to-date
[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] sh
[2023-06-06T14:59:54.576Z] + grep ^version= gradle.properties
[2023-06-06T14:59:54.576Z] + cut -d= -f 2
[Pipeline] dir
[2023-06-06T14:59:55.259Z] Running in 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/quickstart
[Pipeline] {
[Pipeline] sh
[2023-06-06T14:59:57.384Z] + mvn clean install -Dgpg.skip
[2023-06-06T15:00:00.005Z] [INFO] Scanning for projects...
[2023-06-06T15:00:00.005Z] [INFO] 

[2023-06-06T15:00:00.005Z] [INFO] Reactor Build Order:
[2023-06-06T15:00:00.005Z] [INFO] 
[2023-06-06T15:00:00.005Z] [INFO] Kafka Streams :: Quickstart   
 [pom]
[2023-06-06T15:00:00.005Z] [INFO] streams-quickstart-java   
 [maven-archetype]
[2023-06-06T15:00:00.005Z] [INFO] 
[2023-06-06T15:00:00.005Z] [INFO] < 
org.apache.kafka:streams-quickstart >-
[2023-06-06T15:00:00.005Z] [INFO] Building Kafka Streams :: Quickstart 
3.6.0-SNAPSHOT[1/2]
[2023-06-06T15:00:00.005Z] [INFO]   from pom.xml
[2023-06-06T15:00:00.005Z] [INFO] [ pom 
]-
[2023-06-06T15:00:00.939Z] [INFO] 
[2023-06-06T15:00:00.939Z] [INFO] --- clean:3.0.0:clean (default-clean) @ 
streams-quickstart ---
[2023-06-06T15:00:00.939Z] [INFO] 
[2023-06-06T15:00:00.939Z] [INFO] --- remote-resources:1.5:process 
(process-resource-bundles) @ streams-quickstart ---
[2023-06-06T15:00:01.872Z] [INFO] 
[2023-06-06T15:00:01.872Z] [INFO] --- site:3.5.1:attach-descriptor 
(attach-descriptor) @ streams-quickstart ---
[2023-06-06T15:00:02.805Z] [INFO] 
[2023-06-06T15:00:02.805Z] [INFO] --- gpg:1.6:sign (sign-artifacts) @ 
streams-quickstart ---
[2023-06-06T15:00:02.805Z] [INFO] 
[2023-06-06T15:00:02.805Z] [INFO] --- install:2.5.2:install (default-install) @ 
streams-quickstart ---
[2023-06-06T15:00:02.805Z] [INFO] Installing 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/quickstart/pom.xml
 to 
/home/jenkins/.m2/repository/org/apache/kafka/streams-quickstart/3.6.0-SNAPSHOT/streams-quickstart-3.6.0-SNAPSHOT.pom
[2023-06-06T15:00:02.805Z] [INFO] 
[2023-06-06T15:00:02.805Z] [INFO] --< 
org.apache.kafka:streams-quickstart-java >--
[2023-06-06T15:00:02.805Z] [INFO] Building streams-quickstart-java 
3.6.0-SNAPSHOT[2/2]
[2023-06-06T15:00:02.805Z] [INFO]   from java/pom.xml
[2023-06-06T15:00:02.805Z] [INFO] --[ maven-archetype 
]---
[2023-06-06T15:00:02.805Z] [INFO] 
[2023-06-06T15:00:02.805Z] [INFO] --- clean:3.0.0:clean (default-clean) @ 
streams-quickstart-java ---
[2023-06-06T15:00:02.805Z] [INFO] 
[2023-06-06T15:00:02.805Z] [INFO] --- remote-resources:1.5:process 
(process-resource-bundles) @ streams-quickstart-java ---
[2023-06-06T15:00:02.805Z] [INFO] 
[2023-06-06T15:00:02.805Z] [INFO] --- resources:2.7:resources 
(default-resources) @ streams-quickstart-java ---
[2023-06-06T15:00:02.805Z] 

[DISCUSS] KIP-936: Throttle number of active PIDs

[Omnia Ibrahim]

Hi everyone,
I want to start the discussion of the KIP-936 to throttle the number of
active PIDs per KafkaPrincipal. The proposal is here
https://cwiki.apache.org/confluence/display/KAFKA/KIP-936%3A+Throttle+number+of+active+PIDs


Thanks for your time and feedback.
Omnia

[jira] [Created] (KAFKA-15063) Throttle number of active PIDs

[Omnia Ibrahim (Jira)]

Omnia Ibrahim created KAFKA-15063:
-

 Summary: Throttle number of active PIDs
 Key: KAFKA-15063
 URL: https://issues.apache.org/jira/browse/KAFKA-15063
 Project: Kafka
  Issue Type: New Feature
  Components: core, producer 
Affects Versions: 3.4.0, 3.2.0, 3.0.0, 3.1.0, 2.8.0, 3.3
Reporter: Omnia Ibrahim


{color:#172b4d}Ticket to track to track KIP-936. Since KIP-679 
i{color:#172b4d}dempotent{color} {color:#172b4d}producers became the default in 
Kafka {color:#172b4d}as a result of this all producer instances will be 
assigned PID. The increase of number of PIDs stored in Kafka brokers by 
{color}{{ProducerStateManager}}{color:#172b4d} exposes the broker to OOM errors 
if it has a high number of producers, rogue or misconfigured client(s).{color} 
{color}{color}

{color:#172b4d}{color:#172b4d}{color:#172b4d}{color:#172b4d}{color:#172b4d}{color:#172b4d}The
 broker is still exposed to OOM{color}{color}{color} even after KIP-854 
introduced a separated config to expire PID from transaction IDs if there is 
high number of PID before {color}{{producer.id.expiration.ms}}{color:#172b4d} 
is exceeded. 
{color}{color}{color}

As a result of this the broker will keep experincing OOM and become offline. 
The only way to recover from this is to increase the heap.  

 

{color:#172b4d}KIP-936 is proposing throttling number of PIDs per 
KafkaPrincipal {color}

{color:#172b4d}See 
[https://cwiki.apache.org/confluence/display/KAFKA/KIP-936%3A+Throttle+number+of+active+PIDs]
 {color}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-934: Add DeleteTopicPolicy

[Jorge Esteban Quilcate Otoya]

Thank Colin.

> One last note: if we do this, we should pass the UUID of the topic as
well as the name.

Sure, adding it to the KIP.

> On the minus side, the stated use-case (preventing deletion of
__consumer_offsets) seems easy to solve via ACLs. The CreateTopics case is
different... it's not as easy to solve via ACLs because people wanted to
enforcce specific topic names or conventions, beyond what ACLs could
provide.
> So it would be good to understand a bit more about why ACLs are not a
better solution than deletion policies.

Thanks for highlight this. I elaborated on the motivation a bit further.
I agree ACLs are a main tool to protect against unwanted topic deletion;
but even when proper users are authorized, it may be a human error to
request a topic deletion. So, in this case, policies act as a complement to
ACLs when topic deletion wants to be blocked.

Looking forward to your feedback.

Many thanks,
Jorge.

On Fri, 2 Jun 2023 at 22:24, Colin McCabe  wrote:

> Hi Jorge,
>
> On the plus side, the change is small and pretty easy to support.
>
> On the minus side, the stated use-case (preventing deletion of
> __consumer_offsets) seems easy to solve via ACLs. The CreateTopics case is
> different... it's not as easy to solve via ACLs because people wanted to
> enforcce specific topic names or conventions, beyond what ACLs could
> provide.
>
> So it would be good to understand a bit more about why ACLs are not a
> better solution than deletion policies.
>
> One last note: if we do this, we should pass the UUID of the topic as well
> as the name.
>
> best,
> Colin
>
>
> On Mon, May 22, 2023, at 09:18, Jorge Esteban Quilcate Otoya wrote:
> > Hi everyone,
> >
> > I'd like to start a discussion for KIP-934 <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-934%3A+Add+DeleteTopicPolicy
> >
> > which proposes adding a new policy for when deleting topics.
> >
> > There have been related KIPs in the past that haven't been accepted and
> > seem retired/abandoned as outlined in the motivation.
> > The scope of this KIP intends to be more specific to get most of the
> > benefits from previous discussions; and if previous KIPs are resurrected,
> > should still be possible to do it if this one is adopted.
> >
> > Looking forward to your feedback!
> >
> > Thanks,
> > Jorge.
>

Re: [DISCUSS] KIP-918: MM2 Topic And Group Listener

[Chris Egerton]

Hi Daniel,

Thanks for the KIP! I see the value in exposing information on replicated
topics and groups. For one, it matches a similar feature added to Kafka
Connect in KIP-558 [1], where we started tracking the set of topics that
connectors interacted with over their lifetime. And there's also the use
case you provided about identifying the provenance of topics replicated
with the identity replication policy (or really, any policy that doesn't
preserve information about the source cluster). Finally, it seems like a
decent debugging aid for prototyping and initially standing up MM2
clusters, and a liveness check for existing ones.

Here are my thoughts so far:

1. I know that MM2 has a lot of pluggable interfaces already but I'm always
a little hesitant to introduce another one. One alternative could be to add
new metrics for the sets of replicated topics and groups. Users can already
implement pluggable metrics reporters [2], which could be a substitute for
the listener interface proposed in the KIP.

2. Is the intention to provide the listener with the total current set of
replicated groups and topics every time that set is computed? Or is the
listener given the complete set the first time and a delta other times?
Based on the type signatures of the interface methods I'm guessing it's the
former, but based on the names (which use the word "changed") it seems like
the latter. If we use complete sets, I think "refreshed" or "computed" may
be better as suffixes, or we could possibly use "replicated" as a prefix
("replicatedTopics", "replicatedGroups").

[1] -
https://cwiki.apache.org/confluence/display/KAFKA/KIP-558%3A+Track+the+set+of+actively+used+topics+by+connectors+in+Kafka+Connect
[2] -
https://kafka.apache.org/34/javadoc/org/apache/kafka/common/metrics/MetricsReporter.html

Cheers,

Chris

On Fri, Apr 21, 2023 at 9:00 AM Dániel Urbán  wrote:

> Thanks for the comments Viktor.
> 1. My original motivation was IdentityReplicationPolicy based monitoring.
> The current MirrorClient implementation cannot list the replica topics of
> the target cluster. I think relying on the topic-partition level metrics is
> a complex solution. Instead, I would like to make it simple to collect all
> the replicated topics of a flow, without relying on the name of the topics.
> Then, I simply tried to generalize the approach.
> 2. Checkpoint metrics are reported per (group, topic, partition), it means
> that there is no metric associated with a group. If a filter picks up a
> group, but the group doesn't have committed offsets for any of the
> replicated partitions, there is no metric to be eagerly registered. This is
> a difference between how topic replication and group checkpointing works -
> empty topics are still picked up for partition creation and to consume from
> them. Groups are only picked up if they have committed offsets already.
> 3. Not exactly sure what is the added value of listing all
> topic-partitions, but that information is available where the filtering
> happens. For groups, we don't have anything else besides the group name, so
> we cannot really provide more info at that point without significantly
> changing the refresh group logic.
>
> Thanks,
> Daniel
>
> Viktor Somogyi-Vass  ezt írta
> (időpont: 2023. ápr. 21., P, 11:43):
>
> > Hi all,
> >
> > A couple of comments:
> > 1) Regarding the motivation: is the motivation simply monitoring related
> or
> > are there any other reasons to this?
> > 2) Can we change monitoring to be identical to filters, so that what is
> > actively filtered, we monitor exactly those topics and groups? (So group
> > metrics aren't added lazily when a checkpoint is created but when the
> > filter is changed.)
> > 3) Not sure if we want to widen the scope but since these are interfaces
> > I'd use TopicPartition and some kind of GroupDescription classes (not
> sure
> > if the latter exists) instead of Strings. If later on we'll need extra
> > properties for these then it can be added on easier.
> >
> > Best,
> > Viktor
> >
> > On Wed, Apr 19, 2023 at 1:42 PM Dániel Urbán 
> > wrote:
> >
> > > I wouldn't really include a non-existent group (same as we don't care
> > about
> > > a non-existent topic), that doesn't really matter.
> > > I think having an existing group which doesn't have an offset to
> > checkpoint
> > > is equivalent to a topic having no records to replicate from the
> > monitoring
> > > perspective.
> > >
> > > I think the precise way to put it is to monitor the topics and groups
> > > picked up by the filtering logic of MM2. "The list currently
> replicated"
> > is
> > > not a good definition, as an empty topic would still be interesting for
> > > monitoring purposes, even if there is no message to replicate.
> > > I think the core motivation is to capture the output of the
> > > TopicFilter/GroupFilter + the extra, built-in logic of MM2 related to
> > > filtering (e.g. internal topics are never replicated, the heartbeats
> > topics
> > > are always 

Re: [DISCUSS] Regarding Old PRs

[Mickael Maison]

Hi Josep,

Thanks for looking into this. This is clearly one aspect where we need
to improve.

We had a similar initiative last year
(https://lists.apache.org/thread/66yj9m6tcyz8zqb3lqlbnr386bqwsopt) and
we closed many PRs. Unfortunately we did not follow up with a process
or automation and we are back to the same situation.

Manually reviewing all these PRs is a huge task, so I think we should
at least partially automate it. I'm not sure if we should manually
review the oldest PRs (pre 2020). There's surely many interesting
things but I wonder if we should instead focus on the more recent ones
as they have a higher chance of 1) still making sense, 2) getting
updates from their authors, 3) needing less rebasing. If something has
been broken since 2016 but we never bothered to fix the PR it means it
can't be anything critical!

Finally as Colin mentioned, it looks like a non negligible chunk of
stale PRs comes from committers and regular contributors. So I'd
suggest we each try to clean our own backlog too.

I wonder if we also need to do something in JIRA. Querying for
unresolved tickets returns over 4000 items. Considering we're not
quite at KAFKA-15000 yet, that's a lot.

Thanks,
Mickael


On Tue, Jun 6, 2023 at 11:35 AM Josep Prat  wrote:
>
> Hi Devs,
> I would say we can split the problem in 2.
>
> Waiting for Author feedback:
> We could have a bot that would ping authors for the cases where we have PRs
> that are stalled and have either:
> - Merge conflict
> - Unaddressed reviews
>
> Waiting for reviewers:
> For the PRs where we have no reviewers and there are no conflicts, I think
> we would need some human interaction to determine modules (maybe this can
> be automated) and ping people who could review.
>
> What do you think?
>
> Best,
>
> On Tue, Jun 6, 2023 at 11:30 AM Josep Prat  wrote:
>
> > Hi Nikolay,
> >
> > With a bot it will be complicated to determine what to do when the PR
> > author is waiting for a reviewer. If a person goes over them, can check if
> > they are waiting for reviews and tag the PR accordingly and maybe ping a
> > maintainer.
> > If you look at my last email I described a flow (but AFAIU it will work
> > only if a human executes it) where the situation you point out would be
> > covered.
> >
> > ———
> > Josep Prat
> >
> > Aiven Deutschland GmbH
> >
> > Alexanderufer 3-7, 10117 Berlin
> >
> > Amtsgericht Charlottenburg, HRB 209739 B
> >
> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> >
> > m: +491715557497
> >
> > w: aiven.io
> >
> > e: josep.p...@aiven.io
> >
> > On Tue, Jun 6, 2023, 11:20 Николай Ижиков  wrote:
> >
> >> Hello.
> >>
> >> What is actions of contributor if no feedback given? [1], [2]
> >>
> >> [1] https://github.com/apache/kafka/pull/13278
> >> [2] https://github.com/apache/kafka/pull/13247
> >>
> >> > 2 июня 2023 г., в 23:38, David Arthur  написал(а):
> >> >
> >> > I think this is a great idea. If we don’t want the auto-close
> >> > functionality, we can set it to -1
> >> >
> >> > I realize this isn’t a vote, but I’m +1 on this
> >> >
> >> > -David
> >> >
> >> > On Fri, Jun 2, 2023 at 15:34 Colin McCabe  wrote:
> >> >
> >> >> That should read "30 days without activity"
> >> >>
> >> >> (I am assuming we have the ability to determine when a PR was last
> >> updated
> >> >> on GH)
> >> >>
> >> >> best,
> >> >> Colin
> >> >>
> >> >> On Fri, Jun 2, 2023, at 12:32, Colin McCabe wrote:
> >> >>> Hi all,
> >> >>>
> >> >>> Looking at GitHub, I have a bunch of Kafka PRs of my own that I've
> >> >>> allowed to become stale, and I guess are pushing up these numbers!
> >> >>> Overall I support the goal of putting a time limit on PRs, just so
> >> that
> >> >>> we can focus our review bandwidth.
> >> >>>
> >> >>> It may be best to start with a simple approach where we mark PRs as
> >> >>> stale after 30 days and email the submitter at that time. And then
> >> >>> delete after 60 days. (Of course the exact time periods might be
> >> >>> something gother than 30/60 but this is just an initial suggestion)
> >> >>>
> >> >>> best,
> >> >>> Colin
> >> >>>
> >> >>>
> >> >>> On Fri, Jun 2, 2023, at 00:37, Josep Prat wrote:
> >>  Hi all,
> >> 
> >>  I want to say that in my experience, I always felt better as a
> >> >> contributor
> >>  when a person told me something than when a bot did. That being said,
> >> >> I'm
> >>  not against bots, and I think they might be a great solution once we
> >> >> have a
> >>  manageable number of open PRs.
> >> 
> >>  Another great question that adding a bot poses is types of staleness
> >>  detection. How do we distinguish between staleness from the author's
> >> >> side
> >>  or from the lack of reviewers/maintainers' side? That's why I started
> >> >> with
> >>  a human approach to be able to distinguish between these 2 cases.
> >> Both
> >>  situations should have different messages and actually different
> >> >> intended
> >>  receivers. In case of staleness because of author 

Re: [DISCUSS] solutions for broker OOM caused by many producer IDs

[Omnia Ibrahim]

Thanks, Luke for the feedback

1. how do we store value in bloom filter? It's unclear from this KIP that
> what we store inside bloom filter, and how we throttle them.
> My understanding is, we have a map with key = kafkaPrinciple, and value =
> PID for each bloom filter.
> And when new PID created for a userA, we update the map to add PID into
> the cache value (i.e. the bloom filter)
> When the window passed half of the time, we created another bloom filter,
> and this time, when new PID comes, we check if this new PID existed in
> previous bloom filter, if not, we add into the new bloom filter. And in the
> meantime, we track the "new created" count (filtered by previous bloom
> filter) for throttling the users.
> Is my understanding correct?
>

Not quite what am proposing. I'm proposing

   - a cache layer to be used only for checking if we encountered the PID
   before or not for a given KafkaPrincipal. Not as a counter.
   - if the cache layer doesn't contain the PIDs I'll increment the metric
   sensor using Sensor::record (the sensor will be created during the initial
   interaction with this KafkaPrincipal). Sensor::record fails with
   QuotaViolationException when we reach the max of the sensor.
   - If incrementing the sensor didn't fail with QuotaViolationException
   then I'll add the PID to the cache for the next time

To achieve this I'm proposing this the cache layer will be represented as a
"cachedMap = Map>".
I wrapped "Map>" into
"TimedControlledBloomFilter" where we decide which bloom filter to write
to, which bloomfilter to delete, etc.


   1. When we encounter the producer for the first time,


   - the new quota manager will create the sensor and update its value.
   - Then it will update the cache with this PID for the next time.


   1. The cache will create an entry for the user in the cachedMap
  - The cache will be like this
 - Map { "UserA" -> TimedBloomFilter {
   bloom_filter_1_create_timestamp
-> bloom_filter_1
 }
}
 2. the PID will be added to bloom_filter_1

2. If the producer tries to produce with the same PID the next time; the
quota manager will not update the sensor or the cache. And will not
throttle the user.
3. However, if the producer tries to produce with a new PID, it will be
added to bloom_filter_1 as long as we are within the first half of
producer.id.quota.window.size.seconds
4. If the producer sends any new PIDs after the first half of
producer.id.quota.window.size.seconds, we will create a new bloom filter to
store the PIDs from the next half of the window

   - The cache will be like this
   - Map { "UserA" -> TimedBloomFilter {
 bloom_filter_1_create_timestamp ->
  bloom_filter_1
 bloom_filter_2_create_timestamp ->
  bloom_filter_2
   }
  }
   - All PIDs from this point until the end of this window will be added to
   bloom_filter_2.
   - And both bloom_filter_1 and bloom_filter_2 will be used for checking
   if we came across PID before or not.

5. a scheduler will run in the background to delete any bloom filter with
create_timestamp >= producer.id.quota.window.size.seconds from
TimedBloomFilter automatically
6. If the user stopped producing and all its bloom filters got deleted by
the scheduler the user entry will be removed from the cachedMap.

I updated the KIP to add more clarification to this caching layer.

2. what will user get when they can't allocate new PID due to throttling?
>

The client will get `QuotaViolationException` similar to  ClientQuotaManager
https://cwiki.apache.org/confluence/display/KAFKA/KIP-936:+Throttle+number+of+active+PIDs#KIP936:ThrottlenumberofactivePIDs-ClientErrors

3. This config: producer.id.quota.window.num is unclear to me?
>

`quota.window.num` is a standard config between all Kafka quota types. I am
re-using the same description and default value in the existing Kafka
codebase (am not a fan of the description as it's not clear). The sample
here is referring to the sliding time window. Kafka Quotas keeps 10 in
memory + the current window so in total they are 11.

> Finally, I think this KIP is good to have an official KIP discuss thread
for community review.
I will open the official KIP discussion today.

Thanks
Omnia

On Tue, Jun 6, 2023 at 10:19 AM Luke Chen  wrote:

> Hi Omnia,
>
> I finally got time to check this KIP.
> Thanks for putting all this together.
>
> Questions:
> 1. how do we store value in bloom filter? It's unclear from this KIP that
> what we store inside bloom filter, and how we throttle them.
> My understanding is, we have a map with key = kafkaPrinciple, and value =
> PID for each bloom filter.
> And when new PID created for a userA, we update the map to add PID into
> the cache value (i.e. the bloom filter)
> When the window passed half of 

Re: [DISCUSS] KIP-935: Extend AlterConfigPolicy with existing configurations

[Jorge Esteban Quilcate Otoya]

Thanks Colin.

> I would suggest renaming the "configs" parameter to "proposedConfigs," in
both the new and old RequestMetadata constructors, to make things clearer.
This would be a binary and API-compatible change in Java.

Sure, fully agree. KIP is updated with this suggestion.

> We should also clarify that if configurations are being proposed for
deletion, they won't appear in proposedConfigs.

Great catch. Wasn't aware of this, but I think it's valuable for the API to
also include the list of configurations to be deleted.
For this, I have extended the `RequestMetadata` type with a list of
proposed configs to delete:

```
class RequestMetadata {

private final ConfigResource resource;
private final Map proposedConfigs;
private final List proposedConfigsToDelete;
private final Map existingConfigs;
```

Looking forward to your feedback.

Cheers,
Jorge.

On Fri, 2 Jun 2023 at 22:42, Colin McCabe  wrote:

> Hi Jorge,
>
> This is a good KIP which addresses a real gap we have today.
>
> I would suggest renaming the "configs" parameter to "proposedConfigs," in
> both the new and old RequestMetadata constructors, to make things clearer.
> This would be a binary and API-compatible change in Java. We should also
> clarify that if configurations are being proposed for deletion, they won't
> appear in proposedConfigs.
>
> best,
> Colin
>
>
> On Tue, May 23, 2023, at 03:03, Christo Lolov wrote:
> > Hello!
> >
> > This proposal will address problems with configuration dependencies
> which I
> > run into very frequently, so I am fully supporting the development of
> this
> > feature!
> >
> > Best,
> > Christo
> >
> > On Mon, 22 May 2023 at 17:18, Jorge Esteban Quilcate Otoya <
> > quilcate.jo...@gmail.com> wrote:
> >
> >> Hi everyone,
> >>
> >> I'd like to start a discussion for KIP-935 <
> >>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-935%3A+Extend+AlterConfigPolicy+with+existing+configurations
> >> >
> >> which proposes extend AlterConfigPolicy with existing configuration to
> >> enable more complex policies.
> >>
> >> There have been related KIPs in the past that haven't been accepted and
> >> seem retired/abandoned as outlined in the motivation.
> >> The scope of this KIP intends to be more specific to get most of the
> >> benefits from previous discussions; and if previous KIPs are
> resurrected,
> >> should still be possible to do it if this one is adopted.
> >>
> >> Looking forward to your feedback!
> >>
> >> Thanks,
> >> Jorge.
> >>
>

Re: [VOTE] KIP-923: Add A Grace Period to Stream Table Join

[John Roesler]

Thanks for the KIP, Walker!

I’m +1 (binding)

Thanks,
John

On Mon, Jun 5, 2023, at 13:39, Victoria Xia wrote:
> Hi Walker,
>
> Thanks for the KIP! Left a clarification question on the discussion thread
> just now but it's about an implementation detail, so I don't think it
> changes anything in this vote thread.
>
> +1 (non-binding)
>
> Cheers,
> Victoria
>
> On Mon, Jun 5, 2023 at 10:23 AM Bill Bejeck  wrote:
>
>> Hi Walker,
>>
>> Thanks for the KIP.
>>
>> I've caught up on the discussion thread and I'm satisfied with all
>> responses.
>>
>> +1(binding)
>>
>> -Bill
>>
>> On Mon, Jun 5, 2023 at 10:20 AM Bruno Cadonna  wrote:
>>
>> > Hi Walker,
>> >
>> > Thank you for the KIP!
>> >
>> > +1 (binding)
>> >
>> > Best,
>> > Bruno
>> >
>> > On 24.05.23 23:00, Walker Carlson wrote:
>> > > Hello everybody,
>> > >
>> > > I'm opening the vote on KIP-923 here
>> > > .
>> > >
>> > > If we have more to discus please continue the discussion on the
>> existing
>> > > thread
>> https://www.mail-archive.com/dev@kafka.apache.org/msg130657.html
>> > >
>> > > best,
>> > > Walker
>> > >
>> >
>>

Re: [VOTE] KIP-938: Add more metrics for measuring KRaft performance

[Divij Vaidya]

"Each metadata version has a corresponding integer in the
MetadataVersion.java file."

Please correct me if I'm wrong, but are you referring to "featureLevel" in
the enum at
https://github.com/apache/kafka/blob/trunk/server-common/src/main/java/org/apache/kafka/server/common/MetadataVersion.java#L45
? Is yes, can we please update the description of the metric to make it
easier for the users to understand this? For example, we can say,
"Represents the current metadata version as an integer value. See
MetadataVersion (hyperlink) for a mapping between string and integer
formats of metadata version".

--
Divij Vaidya



On Tue, Jun 6, 2023 at 1:51 PM Ron Dagostino  wrote:

> Thanks again for the KIP, Colin.  +1 (binding).
>
> Ron
>
> > On Jun 6, 2023, at 7:02 AM, Igor Soarez 
> wrote:
> >
> > Thanks for the KIP.
> >
> > Seems straightforward, LGTM.
> > Non binding +1.
> >
> > --
> > Igor
> >
>

Re: [DISCUSS] KIP-937 Improve Message Timestamp Validation

[Divij Vaidya]

Hi Luke

Thank you for your participation in reviewing this KIP.

#1 Updated the KIP with correct configuration names and hyperlinks.

#2 Yes, the semantics change from a perspective that the difference is
always in the past (or at most 1 hour into the future). Updated the
compatibility section to represent the same. Will update again after #3 is
resolved.

#3 I'd propose to introduce 2 new configs, one is "
log.message.timestamp.before.max.ms", and the other one is "
log.message.timestamp.after.max.ms".
This is a great idea because with your proposal, 1\ we can ensure backward
compatibility (hence safety of this breaking change) in 3.x by keeping the
defaults of these configurations in line with what "
log.message.timestamp.difference.max.ms" provides today 2\ and when 4.x has
modified logic to perform segment rotation based on append time which will
always be available, we can still re-use these configurations to reject
messages based on validation of create timestamps. Mehari, thoughts?

--
Divij Vaidya



On Tue, Jun 6, 2023 at 11:43 AM Luke Chen  wrote:

> Hi Beyene,
>
> Thanks for the KIP.
>
> Questions:
> 1. We don't have "*max.message.time.difference.ms
> *" config, I think you're referring
> to "log.message.timestamp.difference.max.ms"?
> 2. After this KIP, the semantics of "
> log.message.timestamp.difference.max.ms"
> will be changed, right?
> We should also mention it in this KIP, maybe compatibility section?
> And I think the description of "log.message.timestamp.difference.max.ms"
> will also need to be updated.
> 3. Personally I like to see the "TIME_DRIFT_TOLERANCE" to be exposed as a
> config, since after this change, the root issue is still not completed
> resolved.
> After this KIP, the 1 hour later of timestamp can still be appended
> successfully, which might still be an issue for some applications.
> I'd propose to introduce 2 new configs, one is "
> log.message.timestamp.before.max.ms", and the other one is "
> log.message.timestamp.after.max.ms".
> And then we deprecate "log.message.timestamp.difference.max.ms". WDYT?
>
> Thank you.
> Luke
>
> On Tue, Jun 6, 2023 at 8:02 AM Beyene, Mehari 
> wrote:
>
> > Hey Justine and Divij,
> >
> > Thank you for the recommendations.
> > I've made the updates to the KIP and added a new section called "Future
> > Work: Update Message Format to Include Both Client Timestamp and
> LogAppend
> > Timestamp."
> >
> > Please take a look when get some time and let me know if there's anything
> > else you'd like me to address.
> >
> > Thanks,
> > Mehari
> >
> > On 6/5/23, 10:16 AM, "Divij Vaidya"  > divijvaidy...@gmail.com>> wrote:
> >
> >
> > CAUTION: This email originated from outside of the organization. Do not
> > click links or open attachments unless you can confirm the sender and
> know
> > the content is safe.
> >
> >
> >
> >
> >
> >
> > Hey Justine
> >
> >
> > Thank you for bringing this up. We had a discussion earlier in this [1]
> > thread and concluded that bumping up the message version is a very
> > expensive operation. Hence, we want to bundle together a bunch of
> > impactful changes that we will perform on the message version and change
> it
> > in v4.0. We are currently capturing the ideas here [2]. The idea of
> always
> > having a log append time is captured in point 4 in the above wiki of
> ideas.
> >
> >
> > As you suggested, we will add a new section called "future work" and add
> > the idea of two timestamps (& why not do it now) over there. Meanwhile,
> > does the above explanation answer your question on why not to do it right
> > now?
> >
> >
> > [1] https://lists.apache.org/thread/rxnps10t4vrsor46cx6xdj6t03qqxosh <
> > https://lists.apache.org/thread/rxnps10t4vrsor46cx6xdj6t03qqxosh>
> > [2]
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/ideas+for+kafka+message+format+v.3
> > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/ideas+for+kafka+message+format+v.3
> > >
> >
> >
> >
> >
> > --
> > Divij Vaidya
> >
> >
> >
> >
> >
> >
> > On Mon, Jun 5, 2023 at 6:42 PM Justine Olshan  > lid>
> > wrote:
> >
> >
> > > Hey Mehari,
> > > Thanks for adding that section. I think one other thing folks have
> > > considered is including two timestamps in the message format -- one for
> > the
> > > client side timestamp and one for the server side. Of course, this
> would
> > > require a bump to the message format, and that hasn't happened in a
> > while.
> > > Could we include some information on this approach and why we aren't
> > > pursuing it? I think message format bumps are tricky, but it is worth
> > > calling out that this is also an option.
> > >
> > > Thanks,
> > > Justine
> > >
> > > On Fri, Jun 2, 2023 at 4:51 PM Beyene, Mehari  > lid>
> > > wrote:
> > >
> > > > Hi Justine,
> > > >
> > > > I added a section under Proposed Changes -> Timestamp Validation
> Logic
> > to
> > > > capture how the 

[GitHub] [kafka-site] showuon commented on a diff in pull request #519: MINOR: Add Kafka 3.4.1 documentation and javadoc and download link

[via GitHub]

showuon commented on code in PR #519:
URL: https://github.com/apache/kafka-site/pull/519#discussion_r1219517807


##
34/javadoc/search.js:
##
@@ -1,26 +1,26 @@
 /*
  * Copyright (c) 2015, 2020, Oracle and/or its affiliates. All rights reserved.
- * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
+ * ORACLE PROPRIETARY/CONFIDENTIAL. Use is subject to license terms.
+ *
+ *
+ *
+ *
+ *
+ *
+ *
+ *
+ *
+ *
+ *
+ *
+ *
+ *
+ *
+ *

Review Comment:
   Change reverted in 
https://github.com/apache/kafka-site/pull/519/commits/3b39b2a6b7c459ef9f53a3b5acf9ed1aedc39307



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] showuon commented on a diff in pull request #519: MINOR: Add Kafka 3.4.1 documentation and javadoc and download link

[via GitHub]

showuon commented on code in PR #519:
URL: https://github.com/apache/kafka-site/pull/519#discussion_r1219517284


##
34/javadoc/script.js:
##
@@ -1,26 +1,26 @@
 /*
  * Copyright (c) 2013, 2020, Oracle and/or its affiliates. All rights reserved.
- * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
- *
- * This code is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License version 2 only, as
- * published by the Free Software Foundation.  Oracle designates this
- * particular file as subject to the "Classpath" exception as provided
- * by Oracle in the LICENSE file that accompanied this code.
- *
- * This code is distributed in the hope that it will be useful, but WITHOUT
- * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
- * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
- * version 2 for more details (a copy is included in the LICENSE file that
- * accompanied this code).
- *
- * You should have received a copy of the GNU General Public License version
- * 2 along with this work; if not, write to the Free Software Foundation,
- * Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA.
- *
- * Please contact Oracle, 500 Oracle Parkway, Redwood Shores, CA 94065 USA
- * or visit www.oracle.com if you need additional information or have any
- * questions.
+ * ORACLE PROPRIETARY/CONFIDENTIAL. Use is subject to license terms.
+ *
+ *
+ *
+ *
+ *
+ *
+ *
+ *
+ *
+ *
+ *
+ *
+ *
+ *
+ *
+ *
+ *
+ *
+ *
+ *

Review Comment:
   Thanks @jlprat for pointing this out. I don't know why the generated output 
looks like this. I found in [2.0 
doc](https://github.com/apache/kafka-site/blob/asf-site/20/javadoc/script.js), 
the result also looks like this. But I think we should be consistent with 
recent builds, i.e.  
[3.5](https://github.com/apache/kafka-site/blob/asf-site/35/javadoc/script.js), 
which uses original license statement. I'll revert that change. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] mimaison merged pull request #518: MINOR: Update upgrade to 3.5 section

[via GitHub]

mimaison merged PR #518:
URL: https://github.com/apache/kafka-site/pull/518


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [VOTE] KIP-938: Add more metrics for measuring KRaft performance

[Ron Dagostino]

Thanks again for the KIP, Colin.  +1 (binding).

Ron

> On Jun 6, 2023, at 7:02 AM, Igor Soarez  wrote:
> 
> Thanks for the KIP.
> 
> Seems straightforward, LGTM.
> Non binding +1.
> 
> --
> Igor
> 

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1900

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 384548 lines...]
[2023-06-06T11:22:41.332Z] 
[2023-06-06T11:22:41.332Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > KStreamAggregationDedupIntegrationTest > 
shouldReduce(TestInfo) PASSED
[2023-06-06T11:22:41.332Z] 
[2023-06-06T11:22:41.332Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > KStreamAggregationDedupIntegrationTest > 
shouldGroupByKey(TestInfo) STARTED
[2023-06-06T11:22:45.942Z] 
[2023-06-06T11:22:45.942Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > KStreamAggregationDedupIntegrationTest > 
shouldGroupByKey(TestInfo) PASSED
[2023-06-06T11:22:45.942Z] 
[2023-06-06T11:22:45.942Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > KStreamAggregationDedupIntegrationTest > 
shouldReduceWindowed(TestInfo) STARTED
[2023-06-06T11:22:49.699Z] 
[2023-06-06T11:22:49.699Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > KStreamAggregationDedupIntegrationTest > 
shouldReduceWindowed(TestInfo) PASSED
[2023-06-06T11:22:53.294Z] 
[2023-06-06T11:22:53.295Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > KStreamKStreamIntegrationTest > shouldOuterJoin() STARTED
[2023-06-06T11:22:59.242Z] 
[2023-06-06T11:22:59.242Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > KStreamKStreamIntegrationTest > shouldOuterJoin() PASSED
[2023-06-06T11:23:01.886Z] 
[2023-06-06T11:23:01.886Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > KTableSourceTopicRestartIntegrationTest > 
shouldRestoreAndProgressWhenTopicNotWrittenToDuringRestoration() STARTED
[2023-06-06T11:23:05.482Z] 
[2023-06-06T11:23:05.482Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > KTableSourceTopicRestartIntegrationTest > 
shouldRestoreAndProgressWhenTopicNotWrittenToDuringRestoration() PASSED
[2023-06-06T11:23:05.482Z] 
[2023-06-06T11:23:05.482Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > KTableSourceTopicRestartIntegrationTest > 
shouldRestoreAndProgressWhenTopicWrittenToDuringRestorationWithEosAlphaEnabled()
 STARTED
[2023-06-06T11:23:10.287Z] 
[2023-06-06T11:23:10.287Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > KTableSourceTopicRestartIntegrationTest > 
shouldRestoreAndProgressWhenTopicWrittenToDuringRestorationWithEosAlphaEnabled()
 PASSED
[2023-06-06T11:23:10.287Z] 
[2023-06-06T11:23:10.287Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > KTableSourceTopicRestartIntegrationTest > 
shouldRestoreAndProgressWhenTopicWrittenToDuringRestorationWithEosDisabled() 
STARTED
[2023-06-06T11:23:13.885Z] 
[2023-06-06T11:23:13.885Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > KTableSourceTopicRestartIntegrationTest > 
shouldRestoreAndProgressWhenTopicWrittenToDuringRestorationWithEosDisabled() 
PASSED
[2023-06-06T11:23:13.885Z] 
[2023-06-06T11:23:13.885Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > KTableSourceTopicRestartIntegrationTest > 
shouldRestoreAndProgressWhenTopicWrittenToDuringRestorationWithEosV2Enabled() 
STARTED
[2023-06-06T11:23:17.483Z] 
[2023-06-06T11:23:17.483Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > KTableSourceTopicRestartIntegrationTest > 
shouldRestoreAndProgressWhenTopicWrittenToDuringRestorationWithEosV2Enabled() 
PASSED
[2023-06-06T11:23:20.130Z] 
[2023-06-06T11:23:20.130Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > RestoreIntegrationTest > 
shouldRecycleStateFromStandbyTaskPromotedToActiveTaskAndNotRestore(boolean) > 
[1] true STARTED
[2023-06-06T11:23:29.923Z] 
[2023-06-06T11:23:29.923Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > SmokeTestDriverIntegrationTest > 
shouldWorkWithRebalance(boolean) > [2] false PASSED
[2023-06-06T11:23:30.872Z] 
[2023-06-06T11:23:30.872Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > StoreQueryIntegrationTest > 
shouldQuerySpecificActivePartitionStores() STARTED
[2023-06-06T11:23:35.707Z] 
[2023-06-06T11:23:35.707Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > StoreQueryIntegrationTest > 
shouldQuerySpecificActivePartitionStores() PASSED
[2023-06-06T11:23:35.707Z] 
[2023-06-06T11:23:35.707Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > StoreQueryIntegrationTest > 
shouldFailWithIllegalArgumentExceptionWhenIQPartitionerReturnsMultiplePartitions()
 STARTED
[2023-06-06T11:23:37.482Z] 
[2023-06-06T11:23:37.482Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > StoreQueryIntegrationTest > 
shouldFailWithIllegalArgumentExceptionWhenIQPartitionerReturnsMultiplePartitions()
 PASSED
[2023-06-06T11:23:37.482Z] 
[2023-06-06T11:23:37.482Z] 

[GitHub] [kafka-site] jlprat commented on a diff in pull request #519: MINOR: Add Kafka 3.4.1 documentation and javadoc and download link

[via GitHub]

jlprat commented on code in PR #519:
URL: https://github.com/apache/kafka-site/pull/519#discussion_r1219433080


##
34/javadoc/search.js:
##
@@ -1,26 +1,26 @@
 /*
  * Copyright (c) 2015, 2020, Oracle and/or its affiliates. All rights reserved.
- * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
+ * ORACLE PROPRIETARY/CONFIDENTIAL. Use is subject to license terms.
+ *
+ *
+ *
+ *
+ *
+ *
+ *
+ *
+ *
+ *
+ *
+ *
+ *
+ *
+ *
+ *

Review Comment:
   Same as above



##
34/javadoc/script.js:
##
@@ -1,26 +1,26 @@
 /*
  * Copyright (c) 2013, 2020, Oracle and/or its affiliates. All rights reserved.
- * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
- *
- * This code is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License version 2 only, as
- * published by the Free Software Foundation.  Oracle designates this
- * particular file as subject to the "Classpath" exception as provided
- * by Oracle in the LICENSE file that accompanied this code.
- *
- * This code is distributed in the hope that it will be useful, but WITHOUT
- * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
- * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
- * version 2 for more details (a copy is included in the LICENSE file that
- * accompanied this code).
- *
- * You should have received a copy of the GNU General Public License version
- * 2 along with this work; if not, write to the Free Software Foundation,
- * Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA.
- *
- * Please contact Oracle, 500 Oracle Parkway, Redwood Shores, CA 94065 USA
- * or visit www.oracle.com if you need additional information or have any
- * questions.
+ * ORACLE PROPRIETARY/CONFIDENTIAL. Use is subject to license terms.
+ *
+ *
+ *
+ *
+ *
+ *
+ *
+ *
+ *
+ *
+ *
+ *
+ *
+ *
+ *
+ *
+ *
+ *
+ *
+ *

Review Comment:
   I'm surprised about this one, but I'm guessing it's Oracle that changed the 
header on the file.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [VOTE] KIP-938: Add more metrics for measuring KRaft performance

[Igor Soarez]

Thanks for the KIP.

Seems straightforward, LGTM.
Non binding +1.

--
Igor

Re: [VOTE] 3.4.1 RC3

[Yash Mayya]

Hi Prem,

You can follow the instructions here if you wish to unsubscribe from one or
more mailing lists - https://kafka.apache.org/contact

On Tue, Jun 6, 2023 at 2:44 PM Prem Sagar 
wrote:

> Please remove my mail ID from the database.
>
>
> Warm Regards
> K Prem Sagar
> Sr.Manager - Procurements
>  M: + 91 - 9100939886
> p...@pidatacenters.com
>  Amaravati | Bengaluru | Chennai | Delhi | Hyderabad | Kochi | Mumbai
> 
>
> <
> https://pidatacenters.com/news/indias-best-multi-tenant-data-center-service-provider-dcd/
> >
> 
> 
> 
>  
>
>
> On Mon, Jun 5, 2023 at 2:47 PM Josep Prat 
> wrote:
>
> > Hi Luke,
> >
> > Thanks a lot for the patience you had for this release!
> >
> > @Prem you are probably subscribed to either the user or dev mailing list
> > for Apache Kafka, this is why you are receiving these emails.
> >
> > Best,
> >
> > On Mon, Jun 5, 2023 at 10:32 AM Prem Sagar  > .invalid>
> > wrote:
> >
> > > Why this mail is marked to me ?
> > >
> > >
> > > Warm Regards
> > > K Prem Sagar
> > > Sr.Manager - Procurements
> > >  M: + 91 - 9100939886
> > > p...@pidatacenters.com
> > >  Amaravati | Bengaluru | Chennai | Delhi | Hyderabad | Kochi | Mumbai
> > > 
> > >
> > > <
> > >
> >
> https://pidatacenters.com/news/indias-best-multi-tenant-data-center-service-provider-dcd/
> > > >
> > > 
> > > 
> > > 
> > >  
> > >
> > >
> > > On Mon, Jun 5, 2023 at 1:47 PM Luke Chen  wrote:
> > >
> > > > Hi Tom,
> > > >
> > > > Thanks for the vote.
> > > > I've re-run the 3.4 jenkins build, and the
> > > > `DynamicBrokerReconfigurationTest.testAdvertisedListenerUpdate` test
> > > still
> > > > pass.
> > > > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.4/142/
> > > >
> > > > And now, I've got:
> > > > Binding +1 PMC votes:
> > > > * Chris Egerton
> > > > * Mickael Maison
> > > > * Tom Bentley
> > > >
> > > > Non-binding votes:
> > > > * Federico Valeri
> > > > * Jakub Scholz
> > > > * Josep Prat
> > > >
> > > > I will close this vote thread and go ahead to complete the release
> > > process.
> > > >
> > > >
> > > > Thank you.
> > > > Luke
> > > >
> > > > On Fri, Jun 2, 2023 at 5:06 PM Josep Prat
>  > >
> > > > wrote:
> > > >
> > > > > Hi Tom,
> > > > > it failed for me a couple of times, I rebooted and things suddenly
> > > > worked.
> > > > > So maybe there was a dangling process holding a port from previous
> > test
> > > > > failures.
> > > > >
> > > > > On Fri, Jun 2, 2023 at 10:52 AM Tom Bentley 
> > > wrote:
> > > > >
> > > > > > Hi Luke,
> > > > > >
> > > > > > Thanks for running the release.
> > > > > >
> > > > > > I've checked signatures, eyeballed the Javadocs, built from
> source
> > > and
> > > > > run
> > > > > > the unit and integration tests.
> > > > > > DynamicBrokerReconfigurationTest.testAdvertisedListenerUpdate
> fails
> > > for
> > > > > me
> > > > > > repeatedly. I opened
> > > https://issues.apache.org/jira/browse/KAFKA-15049
> > > > > for
> > > > > > it since I couldn't find an existing issue for this one. I note
> > that
> > > > > others
> > > > > > seem to have run the integration tests without problems, so I
> don't
> > > > think
> > > > > > this is a blocker. I also did the Kafka, Connect and Streams
> > > > quickstarts.
> > > > > >
> > > > > > +1 binding.
> > > > > >
> > > > > > Tom
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Thu, 1 Jun 2023 at 08:46, Luke Chen 
> wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > Thanks to everyone who has tested and voted for the RC3 so far!
> > > > > > > Currently, I've got 2 binding votes and 3 non-binding votes:
> > > > > > >
> > > > > > > Binding +1 PMC votes:
> > > > > > > * Chris Egerton
> > > > > > > * Mickael Maison
> > > > > > >
> > > > > > > Non-binding votes:
> > > > > > > * Federico Valeri
> > > > > > > * Jakub Scholz
> > > > > > > * Josep Prat
> > > > > > >
> > > > > > > If anyone is available (especially PMC members :)), please help
> > > > verify
> > > > > > the
> > > > > > > RC build.
> > > > > > >
> > > > > > > Thank you.
> > > > > > > Luke
> > > > > > >
> > > > > > > On Wed, May 31, 2023 at 1:53 AM Chris Egerton
> > > >  > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Luke,
> > > > > > > >
> > > > > > > > Many thanks for your continued work on this release!
> > > > > > > >
> > > > > > > > To verify, I:
> > > > > > > > - Built from source using Java 11 with both:
> > > > > > > > - - the 3.4.1-rc3 tag on GitHub
> > > > > > > > - - the kafka-3.4.1-src.tgz artifact from
> > > > > > > > 

[jira] [Created] (KAFKA-15062) Power(ppc64le) support for Kafka

[Vaibhav (Jira)]

Vaibhav created KAFKA-15062:
---

 Summary: Power(ppc64le) support for Kafka
 Key: KAFKA-15062
 URL: https://issues.apache.org/jira/browse/KAFKA-15062
 Project: Kafka
  Issue Type: Task
  Components: build
Reporter: Vaibhav


Support for Power architecture (ppc64le) for apache kafka.

What is IBM Power architecture?
It is a RISC architecture and IBM has recently made its ISA (Instruction Set 
Architecture) opensource and in doing so, they have significantly contributed 
back to the opensource community at large. Many of the pioneers of banking and 
HPC industries today run on ppc64le architecture.

As an ongoing effort to enable open-source projects where Power architecture 
can add value, we are trying to enable kafka on Power.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Partial CI builds - Reducing flakiness with fewer tests

[David Jacot]

Hey Greg,

Thanks for bringing this up.

I am not sure to understand the benefit of the parallele trigger of a
subset of the tests. Indeed, they will be more likely to pass but the
downside is that folks may start to only rely on that signal and commit
without looking at the full test suite. This seems dangerous to me.

However, I agree that we have an issue with our builds. We have way too
many flaky tests. I would rather focus on trying to address this first. If
we can stabilize them, I wonder if we should also enforce a green build to
merge.

Best,
David



On Mon, Jun 5, 2023 at 7:47 PM Greg Harris 
wrote:

> Hey all,
>
> I've been working on test flakiness recently, and I've been trying to
> come up with ways to tackle the issue top-down as well as bottom-up,
> and I'm interested to hear your thoughts on an idea.
>
> In addition to the current full-suite runs, can we in parallel trigger
> a smaller test run which has only a relevant subset of tests? For
> example, if someone is working on one sub-module, the CI would only
> run tests in that module.
>
> I think this would be more likely to pass than the full suite due to
> the fewer tests failing probabilistically, and would improve the
> signal-to-noise ratio of the summary pass/fail marker on GitHub. This
> should also be shorter to execute than the full suite, allowing for
> faster cycle-time than the current full suite encourages.
>
> This would also strengthen the incentive for contributors specializing
> in a module to de-flake tests, as they are rewarded with a tangible
> improvement within their area of the project. Currently, even the
> modules with the most reliable tests receive consistent CI failures
> from other less reliable modules.
>
> I believe this is possible, even if there isn't an off-the-shelf
> solution for it. We can learn of the changed files via a git diff, map
> that to modules containing those files, and then execute the tests
> just for those modules with gradle. GitHub also permits showing
> multiple "checks" so that we can emit both the full-suite and partial
> test results.
>
> Thanks,
> Greg
>

Re: [DISCUSS] KIP-937 Improve Message Timestamp Validation

[Luke Chen]

Hi Beyene,

Thanks for the KIP.

Questions:
1. We don't have "*max.message.time.difference.ms
*" config, I think you're referring
to "log.message.timestamp.difference.max.ms"?
2. After this KIP, the semantics of "log.message.timestamp.difference.max.ms"
will be changed, right?
We should also mention it in this KIP, maybe compatibility section?
And I think the description of "log.message.timestamp.difference.max.ms"
will also need to be updated.
3. Personally I like to see the "TIME_DRIFT_TOLERANCE" to be exposed as a
config, since after this change, the root issue is still not completed
resolved.
After this KIP, the 1 hour later of timestamp can still be appended
successfully, which might still be an issue for some applications.
I'd propose to introduce 2 new configs, one is "
log.message.timestamp.before.max.ms", and the other one is "
log.message.timestamp.after.max.ms".
And then we deprecate "log.message.timestamp.difference.max.ms". WDYT?

Thank you.
Luke

On Tue, Jun 6, 2023 at 8:02 AM Beyene, Mehari 
wrote:

> Hey Justine and Divij,
>
> Thank you for the recommendations.
> I've made the updates to the KIP and added a new section called "Future
> Work: Update Message Format to Include Both Client Timestamp and LogAppend
> Timestamp."
>
> Please take a look when get some time and let me know if there's anything
> else you'd like me to address.
>
> Thanks,
> Mehari
>
> On 6/5/23, 10:16 AM, "Divij Vaidya"  divijvaidy...@gmail.com>> wrote:
>
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
>
>
>
>
>
> Hey Justine
>
>
> Thank you for bringing this up. We had a discussion earlier in this [1]
> thread and concluded that bumping up the message version is a very
> expensive operation. Hence, we want to bundle together a bunch of
> impactful changes that we will perform on the message version and change it
> in v4.0. We are currently capturing the ideas here [2]. The idea of always
> having a log append time is captured in point 4 in the above wiki of ideas.
>
>
> As you suggested, we will add a new section called "future work" and add
> the idea of two timestamps (& why not do it now) over there. Meanwhile,
> does the above explanation answer your question on why not to do it right
> now?
>
>
> [1] https://lists.apache.org/thread/rxnps10t4vrsor46cx6xdj6t03qqxosh <
> https://lists.apache.org/thread/rxnps10t4vrsor46cx6xdj6t03qqxosh>
> [2]
>
> https://cwiki.apache.org/confluence/display/KAFKA/ideas+for+kafka+message+format+v.3
> <
> https://cwiki.apache.org/confluence/display/KAFKA/ideas+for+kafka+message+format+v.3
> >
>
>
>
>
> --
> Divij Vaidya
>
>
>
>
>
>
> On Mon, Jun 5, 2023 at 6:42 PM Justine Olshan  lid>
> wrote:
>
>
> > Hey Mehari,
> > Thanks for adding that section. I think one other thing folks have
> > considered is including two timestamps in the message format -- one for
> the
> > client side timestamp and one for the server side. Of course, this would
> > require a bump to the message format, and that hasn't happened in a
> while.
> > Could we include some information on this approach and why we aren't
> > pursuing it? I think message format bumps are tricky, but it is worth
> > calling out that this is also an option.
> >
> > Thanks,
> > Justine
> >
> > On Fri, Jun 2, 2023 at 4:51 PM Beyene, Mehari  lid>
> > wrote:
> >
> > > Hi Justine,
> > >
> > > I added a section under Proposed Changes -> Timestamp Validation Logic
> to
> > > capture how the INVALID_TIMESTAMP is handled in this KIP.
> > > Please let me know if there are any additional areas you would like me
> to
> > > address.
> > >
> > > Thanks,
> > > Mehari
> > >
> > >
> > >
> > >
> > >
> >
>
>
>
>

Re: [DISCUSS] Regarding Old PRs

[Josep Prat]

Hi Devs,
I would say we can split the problem in 2.

Waiting for Author feedback:
We could have a bot that would ping authors for the cases where we have PRs
that are stalled and have either:
- Merge conflict
- Unaddressed reviews

Waiting for reviewers:
For the PRs where we have no reviewers and there are no conflicts, I think
we would need some human interaction to determine modules (maybe this can
be automated) and ping people who could review.

What do you think?

Best,

On Tue, Jun 6, 2023 at 11:30 AM Josep Prat  wrote:

> Hi Nikolay,
>
> With a bot it will be complicated to determine what to do when the PR
> author is waiting for a reviewer. If a person goes over them, can check if
> they are waiting for reviews and tag the PR accordingly and maybe ping a
> maintainer.
> If you look at my last email I described a flow (but AFAIU it will work
> only if a human executes it) where the situation you point out would be
> covered.
>
> ———
> Josep Prat
>
> Aiven Deutschland GmbH
>
> Alexanderufer 3-7, 10117 Berlin
>
> Amtsgericht Charlottenburg, HRB 209739 B
>
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>
> m: +491715557497
>
> w: aiven.io
>
> e: josep.p...@aiven.io
>
> On Tue, Jun 6, 2023, 11:20 Николай Ижиков  wrote:
>
>> Hello.
>>
>> What is actions of contributor if no feedback given? [1], [2]
>>
>> [1] https://github.com/apache/kafka/pull/13278
>> [2] https://github.com/apache/kafka/pull/13247
>>
>> > 2 июня 2023 г., в 23:38, David Arthur  написал(а):
>> >
>> > I think this is a great idea. If we don’t want the auto-close
>> > functionality, we can set it to -1
>> >
>> > I realize this isn’t a vote, but I’m +1 on this
>> >
>> > -David
>> >
>> > On Fri, Jun 2, 2023 at 15:34 Colin McCabe  wrote:
>> >
>> >> That should read "30 days without activity"
>> >>
>> >> (I am assuming we have the ability to determine when a PR was last
>> updated
>> >> on GH)
>> >>
>> >> best,
>> >> Colin
>> >>
>> >> On Fri, Jun 2, 2023, at 12:32, Colin McCabe wrote:
>> >>> Hi all,
>> >>>
>> >>> Looking at GitHub, I have a bunch of Kafka PRs of my own that I've
>> >>> allowed to become stale, and I guess are pushing up these numbers!
>> >>> Overall I support the goal of putting a time limit on PRs, just so
>> that
>> >>> we can focus our review bandwidth.
>> >>>
>> >>> It may be best to start with a simple approach where we mark PRs as
>> >>> stale after 30 days and email the submitter at that time. And then
>> >>> delete after 60 days. (Of course the exact time periods might be
>> >>> something gother than 30/60 but this is just an initial suggestion)
>> >>>
>> >>> best,
>> >>> Colin
>> >>>
>> >>>
>> >>> On Fri, Jun 2, 2023, at 00:37, Josep Prat wrote:
>>  Hi all,
>> 
>>  I want to say that in my experience, I always felt better as a
>> >> contributor
>>  when a person told me something than when a bot did. That being said,
>> >> I'm
>>  not against bots, and I think they might be a great solution once we
>> >> have a
>>  manageable number of open PRs.
>> 
>>  Another great question that adding a bot poses is types of staleness
>>  detection. How do we distinguish between staleness from the author's
>> >> side
>>  or from the lack of reviewers/maintainers' side? That's why I started
>> >> with
>>  a human approach to be able to distinguish between these 2 cases.
>> Both
>>  situations should have different messages and actually different
>> >> intended
>>  receivers. In case of staleness because of author inactivity, the
>> >> message
>>  should encourage the author to update the PR with the requested
>> changes
>> >> or
>>  resolve the conflicts. But In many cases, PRs are stale because of
>> lack
>> >> of
>>  reviewers. This would need a different message, targeting
>> maintainers.
>> 
>>  Ideally (with bot or not) I believe the process should be as follows:
>>  - Check PRs that are stale.
>>  - See if they have labels, if not add proper labels (connect, core,
>>  clients...)
>>  -  PR has merge conflicts
>>  -- Merge conflicts exist and target files that still exist, ping the
>> >> author
>>  asking for conflict resolution and add some additional label like
>> >> `stale`.
>>  -- Merge conflicts exist and target files that do not exist anymore,
>> let
>>  the author know that this PR is obsolete, label the PR as 'obsolete'
>> and
>>  close the PR.
>>  - PR is mergeable, check whose action is needed (author or reviewers)
>>  -- Author: let the author know that there are pending comments to
>> >> address.
>>  Add some additional label, maybe `stale` again
>>  -- Reviewer: ping some reviewers that have experience or lately
>> touched
>>  this piece of the codebase, add a label `reviewer needed` or
>> something
>>  similar
>>  - PRs that have `stale` label after X days, will be closed.
>> 
>>  Regarding the comments about only committers and collaborators being
>> >> able
>>  

Re: [DISCUSS] Regarding Old PRs

[Josep Prat]

Hi Nikolay,

With a bot it will be complicated to determine what to do when the PR
author is waiting for a reviewer. If a person goes over them, can check if
they are waiting for reviews and tag the PR accordingly and maybe ping a
maintainer.
If you look at my last email I described a flow (but AFAIU it will work
only if a human executes it) where the situation you point out would be
covered.

———
Josep Prat

Aiven Deutschland GmbH

Alexanderufer 3-7, 10117 Berlin

Amtsgericht Charlottenburg, HRB 209739 B

Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen

m: +491715557497

w: aiven.io

e: josep.p...@aiven.io

On Tue, Jun 6, 2023, 11:20 Николай Ижиков  wrote:

> Hello.
>
> What is actions of contributor if no feedback given? [1], [2]
>
> [1] https://github.com/apache/kafka/pull/13278
> [2] https://github.com/apache/kafka/pull/13247
>
> > 2 июня 2023 г., в 23:38, David Arthur  написал(а):
> >
> > I think this is a great idea. If we don’t want the auto-close
> > functionality, we can set it to -1
> >
> > I realize this isn’t a vote, but I’m +1 on this
> >
> > -David
> >
> > On Fri, Jun 2, 2023 at 15:34 Colin McCabe  wrote:
> >
> >> That should read "30 days without activity"
> >>
> >> (I am assuming we have the ability to determine when a PR was last
> updated
> >> on GH)
> >>
> >> best,
> >> Colin
> >>
> >> On Fri, Jun 2, 2023, at 12:32, Colin McCabe wrote:
> >>> Hi all,
> >>>
> >>> Looking at GitHub, I have a bunch of Kafka PRs of my own that I've
> >>> allowed to become stale, and I guess are pushing up these numbers!
> >>> Overall I support the goal of putting a time limit on PRs, just so that
> >>> we can focus our review bandwidth.
> >>>
> >>> It may be best to start with a simple approach where we mark PRs as
> >>> stale after 30 days and email the submitter at that time. And then
> >>> delete after 60 days. (Of course the exact time periods might be
> >>> something gother than 30/60 but this is just an initial suggestion)
> >>>
> >>> best,
> >>> Colin
> >>>
> >>>
> >>> On Fri, Jun 2, 2023, at 00:37, Josep Prat wrote:
>  Hi all,
> 
>  I want to say that in my experience, I always felt better as a
> >> contributor
>  when a person told me something than when a bot did. That being said,
> >> I'm
>  not against bots, and I think they might be a great solution once we
> >> have a
>  manageable number of open PRs.
> 
>  Another great question that adding a bot poses is types of staleness
>  detection. How do we distinguish between staleness from the author's
> >> side
>  or from the lack of reviewers/maintainers' side? That's why I started
> >> with
>  a human approach to be able to distinguish between these 2 cases. Both
>  situations should have different messages and actually different
> >> intended
>  receivers. In case of staleness because of author inactivity, the
> >> message
>  should encourage the author to update the PR with the requested
> changes
> >> or
>  resolve the conflicts. But In many cases, PRs are stale because of
> lack
> >> of
>  reviewers. This would need a different message, targeting maintainers.
> 
>  Ideally (with bot or not) I believe the process should be as follows:
>  - Check PRs that are stale.
>  - See if they have labels, if not add proper labels (connect, core,
>  clients...)
>  -  PR has merge conflicts
>  -- Merge conflicts exist and target files that still exist, ping the
> >> author
>  asking for conflict resolution and add some additional label like
> >> `stale`.
>  -- Merge conflicts exist and target files that do not exist anymore,
> let
>  the author know that this PR is obsolete, label the PR as 'obsolete'
> and
>  close the PR.
>  - PR is mergeable, check whose action is needed (author or reviewers)
>  -- Author: let the author know that there are pending comments to
> >> address.
>  Add some additional label, maybe `stale` again
>  -- Reviewer: ping some reviewers that have experience or lately
> touched
>  this piece of the codebase, add a label `reviewer needed` or something
>  similar
>  - PRs that have `stale` label after X days, will be closed.
> 
>  Regarding the comments about only committers and collaborators being
> >> able
>  to label PRs, this is true, not everyone can do this. However, this
> >> could
>  be a great opportunity for the newly appointed contributors to
> exercise
>  their new permissions :)
> 
>  Let me know if it makes sense to you all.
> 
>  Best,
> >>
> > --
> > David Arthur
>
>

Re: [DISCUSS] Regarding Old PRs

[Николай Ижиков]

Hello. 

What is actions of contributor if no feedback given? [1], [2]

[1] https://github.com/apache/kafka/pull/13278
[2] https://github.com/apache/kafka/pull/13247

> 2 июня 2023 г., в 23:38, David Arthur  написал(а):
> 
> I think this is a great idea. If we don’t want the auto-close
> functionality, we can set it to -1
> 
> I realize this isn’t a vote, but I’m +1 on this
> 
> -David
> 
> On Fri, Jun 2, 2023 at 15:34 Colin McCabe  wrote:
> 
>> That should read "30 days without activity"
>> 
>> (I am assuming we have the ability to determine when a PR was last updated
>> on GH)
>> 
>> best,
>> Colin
>> 
>> On Fri, Jun 2, 2023, at 12:32, Colin McCabe wrote:
>>> Hi all,
>>> 
>>> Looking at GitHub, I have a bunch of Kafka PRs of my own that I've
>>> allowed to become stale, and I guess are pushing up these numbers!
>>> Overall I support the goal of putting a time limit on PRs, just so that
>>> we can focus our review bandwidth.
>>> 
>>> It may be best to start with a simple approach where we mark PRs as
>>> stale after 30 days and email the submitter at that time. And then
>>> delete after 60 days. (Of course the exact time periods might be
>>> something gother than 30/60 but this is just an initial suggestion)
>>> 
>>> best,
>>> Colin
>>> 
>>> 
>>> On Fri, Jun 2, 2023, at 00:37, Josep Prat wrote:
 Hi all,
 
 I want to say that in my experience, I always felt better as a
>> contributor
 when a person told me something than when a bot did. That being said,
>> I'm
 not against bots, and I think they might be a great solution once we
>> have a
 manageable number of open PRs.
 
 Another great question that adding a bot poses is types of staleness
 detection. How do we distinguish between staleness from the author's
>> side
 or from the lack of reviewers/maintainers' side? That's why I started
>> with
 a human approach to be able to distinguish between these 2 cases. Both
 situations should have different messages and actually different
>> intended
 receivers. In case of staleness because of author inactivity, the
>> message
 should encourage the author to update the PR with the requested changes
>> or
 resolve the conflicts. But In many cases, PRs are stale because of lack
>> of
 reviewers. This would need a different message, targeting maintainers.
 
 Ideally (with bot or not) I believe the process should be as follows:
 - Check PRs that are stale.
 - See if they have labels, if not add proper labels (connect, core,
 clients...)
 -  PR has merge conflicts
 -- Merge conflicts exist and target files that still exist, ping the
>> author
 asking for conflict resolution and add some additional label like
>> `stale`.
 -- Merge conflicts exist and target files that do not exist anymore, let
 the author know that this PR is obsolete, label the PR as 'obsolete' and
 close the PR.
 - PR is mergeable, check whose action is needed (author or reviewers)
 -- Author: let the author know that there are pending comments to
>> address.
 Add some additional label, maybe `stale` again
 -- Reviewer: ping some reviewers that have experience or lately touched
 this piece of the codebase, add a label `reviewer needed` or something
 similar
 - PRs that have `stale` label after X days, will be closed.
 
 Regarding the comments about only committers and collaborators being
>> able
 to label PRs, this is true, not everyone can do this. However, this
>> could
 be a great opportunity for the newly appointed contributors to exercise
 their new permissions :)
 
 Let me know if it makes sense to you all.
 
 Best,
>> 
> -- 
> David Arthur

Re: [DISCUSS] solutions for broker OOM caused by many producer IDs

[Luke Chen]

Hi Omnia,

I finally got time to check this KIP.
Thanks for putting all this together.

Questions:
1. how do we store value in bloom filter? It's unclear from this KIP that
what we store inside bloom filter, and how we throttle them.
My understanding is, we have a map with key = kafkaPrinciple, and value =
PID for each bloom filter.
And when new PID created for a userA, we update the map to add PID into the
cache value (i.e. the bloom filter)
When the window passed half of the time, we created another bloom filter,
and this time, when new PID comes, we check if this new PID existed in
previous bloom filter, if not, we add into the new bloom filter. And in the
meantime, we track the "new created" count (filtered by previous bloom
filter) for throttling the users.
Is my understanding correct?

2. what will user get when they can't allocate new PID due to throttling?
You might need to address in the KIP.

3. This config: producer.id.quota.window.num is unclear to me?
"The number of samples to retain in memory for alter producer id quotas"
What's the "samples" mean here? Do you mean we only track the top 11
kafkaPrinciple usage each window?

Finally, I think this KIP is good to have an official KIP discuss thread
for community review.
Thanks for the KIP!

Luke




On Mon, Jun 5, 2023 at 11:44 PM Omnia Ibrahim 
wrote:

> Hi Justine, Thanks for having a look
> > One question I have is how will we handle a scenario where potentially
> each new client has a new Kafka Principal? Is that simply not covered by
> throttling?
> if any new client setup a new principal they will be throttled based on
> the throttling for `/config/users/` or
> /config/users/
>
>
> On Wed, May 31, 2023 at 6:50 PM Justine Olshan 
> wrote:
>
>> Hey Omnia,
>>
>> I was doing a bit of snooping (I actually just get updates for the KIP
>> page) and I saw this draft was in progress. I shared it with some of my
>> colleagues as well who I previously discussed the issue with.
>>
>> The initial look was pretty promising to me. I appreciate the detailing
>> of the rejected options since we had quite a few we worked through :)
>>
>> One question I have is how will we handle a scenario where potentially
>> each new client has a new Kafka Principal? Is that simply not covered by
>> throttling?
>>
>> Thanks,
>> Justine
>>
>> On Wed, May 31, 2023 at 10:08 AM Omnia Ibrahim 
>> wrote:
>>
>>> Hi Justine and Luke,
>>>
>>> I started a KIP draft here
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-936%3A+Throttle+number+of+active+PIDs
>>> for a proposal would appreciate it if you could provide any initial
>>> feedback before opening a broader discussion.
>>>
>>> Thanks
>>>
>>> On Wed, Feb 22, 2023 at 4:35 PM Omnia Ibrahim 
>>> wrote:
>>>

 *Hi Justine, *

 *My initial thought of throttling the initProducerId was to get ripped
 off the problem at the source (which creates too many PIDs per client) and
 fail faster but if having this on the produce request level is easier this
 should be fine. I am guessing it will be the same direction as we may
 ClientQuotaManage for Produce throttling with a different quota window than
 `quota.window.size.seconds `. *

 *If this is good as an initial solution I can put start a KIP and see
 what the wider community feels about this. *

 *Also, I noticed that at some point one of us hit "Replay" instead of
 "Replay to All" :)  So here are the previous conversations*

 *On Wed, Feb 15, 2023 at 12:20 AM Justine Olshan >>> > wrote:*

> Hey Omnia,
>
> Thanks for the response. I think I understand your explanations here
> with respect to principal and clientId usage.
>
> For the throttling -- handleInitProducerIdRequest will allocate the ID
> to the producer, but we don't actually store it on the broker or increment
> our metric until the first produce request for that producer is sent (or
> sent again after previously expiring). Would you consider throttling the
> produce request instead? It may be hard to get any metrics from the
> transaction coordinator where the initProducerId request is handled.
>
> Justine


 *On Tue, Feb 14, 2023 at 9:29 AM Omnia Ibrahim >>> > wrote:*

> Hey Justine,
> > If I understand your message correctly, there are issues with
> identifying the source of the rogue clients? So you propose to add a new
> metric for that?
> > And also proposing to throttle based on clientId as a potential
> follow up?
> I want to identify rogue clients by KafkaPrincipal (and/or clientId)
> similarly to how we identify clients in Fetch/Produce/Request
> QuotaManagers. Using KafkaPrincipal should give cluster admin the ability
> to throttle later based on principal which is most likely to be a smaller
> set than clientIds. My initial thought was to add a metrics that represent
> how many InitProducerIDRequest are sent by 

hardware and software requirements to build Apache Kafka

[Prasanna Marathe]

Hi All,
We are working to build Apache Kafka.
We require hardware requirement to build Apache Kafka as CPU core machine, RAM 
and H/W memory.

Request to share info.

Thanks in advance.

With Regards,
Prasanna

Re: [VOTE] 3.4.1 RC3

[Prem Sagar]

Please remove my mail ID from the database.


Warm Regards
K Prem Sagar
Sr.Manager - Procurements
 M: + 91 - 9100939886
p...@pidatacenters.com
 Amaravati | Bengaluru | Chennai | Delhi | Hyderabad | Kochi | Mumbai






 


On Mon, Jun 5, 2023 at 2:47 PM Josep Prat 
wrote:

> Hi Luke,
>
> Thanks a lot for the patience you had for this release!
>
> @Prem you are probably subscribed to either the user or dev mailing list
> for Apache Kafka, this is why you are receiving these emails.
>
> Best,
>
> On Mon, Jun 5, 2023 at 10:32 AM Prem Sagar  .invalid>
> wrote:
>
> > Why this mail is marked to me ?
> >
> >
> > Warm Regards
> > K Prem Sagar
> > Sr.Manager - Procurements
> >  M: + 91 - 9100939886
> > p...@pidatacenters.com
> >  Amaravati | Bengaluru | Chennai | Delhi | Hyderabad | Kochi | Mumbai
> > 
> >
> > <
> >
> https://pidatacenters.com/news/indias-best-multi-tenant-data-center-service-provider-dcd/
> > >
> > 
> > 
> > 
> >  
> >
> >
> > On Mon, Jun 5, 2023 at 1:47 PM Luke Chen  wrote:
> >
> > > Hi Tom,
> > >
> > > Thanks for the vote.
> > > I've re-run the 3.4 jenkins build, and the
> > > `DynamicBrokerReconfigurationTest.testAdvertisedListenerUpdate` test
> > still
> > > pass.
> > > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.4/142/
> > >
> > > And now, I've got:
> > > Binding +1 PMC votes:
> > > * Chris Egerton
> > > * Mickael Maison
> > > * Tom Bentley
> > >
> > > Non-binding votes:
> > > * Federico Valeri
> > > * Jakub Scholz
> > > * Josep Prat
> > >
> > > I will close this vote thread and go ahead to complete the release
> > process.
> > >
> > >
> > > Thank you.
> > > Luke
> > >
> > > On Fri, Jun 2, 2023 at 5:06 PM Josep Prat  >
> > > wrote:
> > >
> > > > Hi Tom,
> > > > it failed for me a couple of times, I rebooted and things suddenly
> > > worked.
> > > > So maybe there was a dangling process holding a port from previous
> test
> > > > failures.
> > > >
> > > > On Fri, Jun 2, 2023 at 10:52 AM Tom Bentley 
> > wrote:
> > > >
> > > > > Hi Luke,
> > > > >
> > > > > Thanks for running the release.
> > > > >
> > > > > I've checked signatures, eyeballed the Javadocs, built from source
> > and
> > > > run
> > > > > the unit and integration tests.
> > > > > DynamicBrokerReconfigurationTest.testAdvertisedListenerUpdate fails
> > for
> > > > me
> > > > > repeatedly. I opened
> > https://issues.apache.org/jira/browse/KAFKA-15049
> > > > for
> > > > > it since I couldn't find an existing issue for this one. I note
> that
> > > > others
> > > > > seem to have run the integration tests without problems, so I don't
> > > think
> > > > > this is a blocker. I also did the Kafka, Connect and Streams
> > > quickstarts.
> > > > >
> > > > > +1 binding.
> > > > >
> > > > > Tom
> > > > >
> > > > >
> > > > >
> > > > > On Thu, 1 Jun 2023 at 08:46, Luke Chen  wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > Thanks to everyone who has tested and voted for the RC3 so far!
> > > > > > Currently, I've got 2 binding votes and 3 non-binding votes:
> > > > > >
> > > > > > Binding +1 PMC votes:
> > > > > > * Chris Egerton
> > > > > > * Mickael Maison
> > > > > >
> > > > > > Non-binding votes:
> > > > > > * Federico Valeri
> > > > > > * Jakub Scholz
> > > > > > * Josep Prat
> > > > > >
> > > > > > If anyone is available (especially PMC members :)), please help
> > > verify
> > > > > the
> > > > > > RC build.
> > > > > >
> > > > > > Thank you.
> > > > > > Luke
> > > > > >
> > > > > > On Wed, May 31, 2023 at 1:53 AM Chris Egerton
> > >  > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Luke,
> > > > > > >
> > > > > > > Many thanks for your continued work on this release!
> > > > > > >
> > > > > > > To verify, I:
> > > > > > > - Built from source using Java 11 with both:
> > > > > > > - - the 3.4.1-rc3 tag on GitHub
> > > > > > > - - the kafka-3.4.1-src.tgz artifact from
> > > > > > > https://home.apache.org/~showuon/kafka-3.4.1-rc3/
> > > > > > > - Checked signatures and checksums
> > > > > > > - Ran the quickstart using the kafka_2.13-3.4.1.tgz artifact
> from
> > > > > > > https://home.apache.org/~showuon/kafka-3.4.1-rc3/ with Java 11
> > and
> > > > > Scala
> > > > > > > 13
> > > > > > > in KRaft mode
> > > > > > > - Ran all unit tests
> > > > > > > - Ran all integration tests for Connect and MM2
> > > > > > >
> > > > > > > +1 (binding)
> > > > > > >
> > > > > > > Cheers,
> > > > > > >
> > > > > > > Chris
> > > > > > >
> > > > > > > On Tue, 

Re: [VOTE] 3.4.1 RC3

[Prem Sagar]

Why this mail is marked to me ?


Warm Regards
K Prem Sagar
Sr.Manager - Procurements
 M: + 91 - 9100939886
p...@pidatacenters.com
 Amaravati | Bengaluru | Chennai | Delhi | Hyderabad | Kochi | Mumbai






 


On Mon, Jun 5, 2023 at 1:47 PM Luke Chen  wrote:

> Hi Tom,
>
> Thanks for the vote.
> I've re-run the 3.4 jenkins build, and the
> `DynamicBrokerReconfigurationTest.testAdvertisedListenerUpdate` test still
> pass.
> https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.4/142/
>
> And now, I've got:
> Binding +1 PMC votes:
> * Chris Egerton
> * Mickael Maison
> * Tom Bentley
>
> Non-binding votes:
> * Federico Valeri
> * Jakub Scholz
> * Josep Prat
>
> I will close this vote thread and go ahead to complete the release process.
>
>
> Thank you.
> Luke
>
> On Fri, Jun 2, 2023 at 5:06 PM Josep Prat 
> wrote:
>
> > Hi Tom,
> > it failed for me a couple of times, I rebooted and things suddenly
> worked.
> > So maybe there was a dangling process holding a port from previous test
> > failures.
> >
> > On Fri, Jun 2, 2023 at 10:52 AM Tom Bentley  wrote:
> >
> > > Hi Luke,
> > >
> > > Thanks for running the release.
> > >
> > > I've checked signatures, eyeballed the Javadocs, built from source and
> > run
> > > the unit and integration tests.
> > > DynamicBrokerReconfigurationTest.testAdvertisedListenerUpdate fails for
> > me
> > > repeatedly. I opened https://issues.apache.org/jira/browse/KAFKA-15049
> > for
> > > it since I couldn't find an existing issue for this one. I note that
> > others
> > > seem to have run the integration tests without problems, so I don't
> think
> > > this is a blocker. I also did the Kafka, Connect and Streams
> quickstarts.
> > >
> > > +1 binding.
> > >
> > > Tom
> > >
> > >
> > >
> > > On Thu, 1 Jun 2023 at 08:46, Luke Chen  wrote:
> > >
> > > > Hi all,
> > > >
> > > > Thanks to everyone who has tested and voted for the RC3 so far!
> > > > Currently, I've got 2 binding votes and 3 non-binding votes:
> > > >
> > > > Binding +1 PMC votes:
> > > > * Chris Egerton
> > > > * Mickael Maison
> > > >
> > > > Non-binding votes:
> > > > * Federico Valeri
> > > > * Jakub Scholz
> > > > * Josep Prat
> > > >
> > > > If anyone is available (especially PMC members :)), please help
> verify
> > > the
> > > > RC build.
> > > >
> > > > Thank you.
> > > > Luke
> > > >
> > > > On Wed, May 31, 2023 at 1:53 AM Chris Egerton
>  > >
> > > > wrote:
> > > >
> > > > > Hi Luke,
> > > > >
> > > > > Many thanks for your continued work on this release!
> > > > >
> > > > > To verify, I:
> > > > > - Built from source using Java 11 with both:
> > > > > - - the 3.4.1-rc3 tag on GitHub
> > > > > - - the kafka-3.4.1-src.tgz artifact from
> > > > > https://home.apache.org/~showuon/kafka-3.4.1-rc3/
> > > > > - Checked signatures and checksums
> > > > > - Ran the quickstart using the kafka_2.13-3.4.1.tgz artifact from
> > > > > https://home.apache.org/~showuon/kafka-3.4.1-rc3/ with Java 11 and
> > > Scala
> > > > > 13
> > > > > in KRaft mode
> > > > > - Ran all unit tests
> > > > > - Ran all integration tests for Connect and MM2
> > > > >
> > > > > +1 (binding)
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Chris
> > > > >
> > > > > On Tue, May 30, 2023 at 11:16 AM Mickael Maison <
> > > > mickael.mai...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Luke,
> > > > > >
> > > > > > I built from source with Java 11 and Scala 2.13 and ran the unit
> > and
> > > > > > integration tests. It took a few retries to get some of them to
> > pass.
> > > > > > I verified signatures and hashes and also ran the zookeeper
> > > quickstart.
> > > > > >
> > > > > > +1 (binding)
> > > > > >
> > > > > > Thanks,
> > > > > > Mickael
> > > > > >
> > > > > > On Sat, May 27, 2023 at 12:58 PM Jakub Scholz 
> > > wrote:
> > > > > > >
> > > > > > > +1 (non-binding) ... I used the staged binaries and Maven
> > artifacts
> > > > to
> > > > > > run
> > > > > > > my tests and all seems to work fine.
> > > > > > >
> > > > > > > Thanks for running the release.
> > > > > > >
> > > > > > > Jakub
> > > > > > >
> > > > > > > On Fri, May 26, 2023 at 9:34 AM Luke Chen 
> > > wrote:
> > > > > > >
> > > > > > > > Hello Kafka users, developers and client-developers,
> > > > > > > >
> > > > > > > > This is the 4th candidate for release of Apache Kafka 3.4.1.
> > > > > > > >
> > > > > > > > This is a bugfix release with several fixes since the release
> > of
> > > > > > 3.4.0. A
> > > > > > > > few of the major issues include:
> > > > > > > > - core
> > > > > > > > KAFKA-14644 <
> https://issues.apache.org/jira/browse/KAFKA-14644
> > >
> > > > > > Process
> > > > > > > > should stop after failure in raft IO thread
> > > > > 

Re: [VOTE] 3.5.0 RC1

[Luke Chen]

Hi Mickael,

I ran the following validation steps:
- Built from source with Java 17 and Scala 2.13
- Signatures and hashes of the artifacts generated
- Navigated through Javadoc including links to JDK classes
- Run the quickstart in KRaft and Zookeeper mode

+1 (binding) from me.
Thanks for running the release.

Luke

On Tue, Jun 6, 2023 at 4:23 AM Josep Prat 
wrote:

> Hi Mickael,
>
> I ran the following validation steps:
> - Built from source with Java 11 and Scala 2.13
> - Signatures and hashes of the artifacts generated
> - Navigated through Javadoc including links to JDK classes
> - Run the unit tests
> - Run integration tests
> - Run the quickstart in KRaft and Zookeeper mode
> -- For KRaft, I looked at the process running Kafka and confirmed that the
> spamming log message is not present anymore ("Generated a metadata delta
> between...")
> - Checked the contents of LICENSE-binary against the lib folder
> -- I found that the LICENSE-binary has a reference to classgraph-4.8.138
> but I fail to find this library in the lib folder. Trying to backtrack when
> it was added, it seems it was done here:
>
> https://github.com/apache/kafka/commit/3a2ac267178c5464e41b94fcbb2dd897212812bd
> but I fail to find this library in the 3.3.1 lib folder (3.3.0 was a dead
> release).
>
> I'm not sure this qualifies as a blocker for the release though, I leave it
> up to you.
>
> Best,
>
> On Mon, Jun 5, 2023 at 3:39 PM Mickael Maison  wrote:
>
> > Hello Kafka users, developers and client-developers,
> >
> > This is the second candidate for release of Apache Kafka 3.5.0. Some
> > of the major features include:
> > - KIP-710: Full support for distributed mode in dedicated MirrorMaker
> > 2.0 clusters
> > - KIP-881: Rack-aware Partition Assignment for Kafka Consumers
> > - KIP-887: Add ConfigProvider to make use of environment variables
> > - KIP-889: Versioned State Stores
> > - KIP-894: Use incrementalAlterConfig for syncing topic configurations
> > - KIP-900: KRaft kafka-storage.sh API additions to support SCRAM for
> > Kafka Brokers
> >
> > Release notes for the 3.5.0 release:
> > https://home.apache.org/~mimaison/kafka-3.5.0-rc1/RELEASE_NOTES.html
> >
> > *** Please download, test and vote by Friday June 9, 5pm PT
> >
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > https://kafka.apache.org/KEYS
> >
> > * Release artifacts to be voted upon (source and binary):
> > https://home.apache.org/~mimaison/kafka-3.5.0-rc1/
> >
> > * Maven artifacts to be voted upon:
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >
> > * Javadoc:
> > https://home.apache.org/~mimaison/kafka-3.5.0-rc1/javadoc/
> >
> > * Tag to be voted upon (off 3.5 branch) is the 3.5.0 tag:
> > https://github.com/apache/kafka/releases/tag/3.5.0-rc1
> >
> > * Documentation:
> > https://kafka.apache.org/35/documentation.html
> >
> > * Protocol:
> > https://kafka.apache.org/35/protocol.html
> >
> > * Successful Jenkins builds for the 3.5 branch:
> > Unit/integration tests: I'm struggling to get all tests to pass in the
> > same build. I'll run a few more builds to ensure each test pass at
> > least once in the CI. All tests passed locally.
> > System tests: The build is still running, I'll send an update once I
> > have the results.
> >
> > Thanks,
> > Mickael
> >
>
>
> --
> [image: Aiven] 
>
> *Josep Prat*
> Open Source Engineering Director, *Aiven*
> josep.p...@aiven.io   |   +491715557497
> aiven.io    |    >
>      <
> https://twitter.com/aiven_io>
> *Aiven Deutschland GmbH*
> Alexanderufer 3-7, 10117 Berlin
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> Amtsgericht Charlottenburg, HRB 209739 B
>

Re: [DISCUSS] KIP-923: Add A Grace Period to Stream Table Join

[Bruno Cadonna]

Hi,

another idea that came to my mind. Instead of using a compacted topic, 
the buffer could use a non-compacted topic and regularly delete records 
before a given offset as Streams does for repartition topics.


Best,
Bruno

On 05.06.23 21:48, Bruno Cadonna wrote:

Hi Victoria,

that is a good point!

I think, the topic needs to be a compacted topic to be able to get rid 
of records that are evicted from the buffer. So the key might be 
something with the key, the timestamp, and a sequence number to 
distinguish between records with the same key and same timestamp.


Just an idea! Maybe Walker comes up with something better.

Best,
Bruno

On 05.06.23 20:38, Victoria Xia wrote:

Hi Walker,

Thanks for the latest updates! The KIP looks great. Just one question 
about

the changelog topic for the join buffer: The KIP says "When a failure
occurs the buffer will try to recover from an OffsetCheckpoint if 
possible.

If not it will reload the buffer from a compacted change-log topic." This
is a new changelog topic that will be introduced specifically for the 
join

buffer, right? Why is the changelog topic compacted? What are the keys? I
am confused because the buffer contains records from the stream-side 
of the

join, for which multiple records with the same key should be treated as
separate updates will all must be tracked in the buffer, rather than
updates which replace each other.

Thanks,
Victoria

On Mon, Jun 5, 2023 at 1:47 AM Bruno Cadonna  wrote:


Hi Walker,

Thanks once more for the updates to the KIP!

Do you also plan to expose metrics for the buffer?

Best,
Bruno

On 02.06.23 17:16, Walker Carlson wrote:

Hello Bruno,

I think this covers your questions. Let me know what you think

2.
We can use a changelog topic. I think we can treat it like any other

store

and recover in the usual manner. Also implementation is on disk

3.
The description is in the public interfaces description. I will copy it
into the proposed changes as well.

This is a bit of an implementation detail that I didn't want to add 
into

the kip, but the record will be added to the buffer to keep the stream

time

consistent, it will just be ejected immediately. If of course if this
causes performance issues we will skip this step and track stream time
separately. I will update the kip to say that stream time advances 
when a

stream record enters the node.

Also, yes, updated.

5.
No there is no difference right now, everything gets processed as it

comes

in and tries to find a record for its time stamp.

Walker

On Fri, Jun 2, 2023 at 6:41 AM Bruno Cadonna  
wrote:



Hi Walker,

Thanks for the updates!

2.
It is still not clear to me how a failure is handled. I do not
understand what you mean by "recover from an OffsetCheckpoint".

My understanding is that the buffer needs to be replicated into its 
own

Kafka topic. The input topic is not enough. The offset of a record is
added to the offsets to commit once the record is streamed through the
subtopology. That means once the record is added to the buffer its
offset is added to the offsets to commit -- independently of 
whether the

record was evicted from the buffer and sent to the join node or not.
Now, let's assume the following scenario
1. a record is read from the input topic and added to the buffer, but
not evicted to be processed by the join node.
2. When the processing of the subtopology finishes the offset of the
record is added to the offsets to commit.
3. A commit happens.
4. A failure happens

After the failure the buffer is empty but the record will not be read
anymore from the input topic since its offset has been already
committed. The record is lost.
One solution to avoid the loss is to recreate the buffer from a
compacted Kafka topic as we do for suppression buffers. I do not 
think,
we need any offset checkpoint here since, we keep the buffer in 
memory,

right? Or do you plan to back the buffer with a persistent store? Even
in that case, a compacted Kafka topic would be needed.


3.
   From the KIP it is still not clear to me what happens if a 
record is

outside of the grace period. I guess the record that falls outside of
the grace period will not be added to the buffer, but will be send to
the join node. Since it is outside of the grace period it will also 
not
increase stream time and it will not trigger an eviction. Also the 
head

of the buffer will not contain a record that needs to be evicted since
the the timestamp of the head record will be within the interval 
stream
time minus grace period. Is this correct? Please add such a 
description

to the KIP.
Furthermore, I think there is a mistake in the text:
"... will dequeue when the record timestamp is greater than stream 
time

plus the grace period". I guess that should be "... will dequeue when
the record timestamp is less than (or equal?) stream time minus the
grace period"


5.
What is the difference between not setting the grace period and 
setting

it to zero? If there is a difference, 

[jira] [Created] (KAFKA-15061) CoordinatorPartitionWriter should reuse buffer

[David Jacot (Jira)]

David Jacot created KAFKA-15061:
---

 Summary: CoordinatorPartitionWriter should reuse buffer
 Key: KAFKA-15061
 URL: https://issues.apache.org/jira/browse/KAFKA-15061
 Project: Kafka
  Issue Type: Sub-task
Reporter: David Jacot
Assignee: David Jacot






--
This message was sent by Atlassian Jira
(v8.20.10#820010)