Re: [kafka-clients] [VOTE] 3.6.0 RC1

[Divij Vaidya]

Hey Satish

My comments about documentation misses from RC0 vote thread [1] are
still not addressed (such as missing metric documentation, formatting
problems etc). Could you please mention why we shouldn't consider them
as blockers to make RC1 as the final release?

[1] https://lists.apache.org/thread/cokoxzd0jtgjtrlxoq7kkzmvpm75381t

On Wed, Sep 20, 2023 at 4:53 PM Satish Duggana  wrote:
>
> Hello Kafka users, developers and client-developers,
>
> This is the second candidate for the release of Apache Kafka 3.6.0. Some of 
> the major features include:
>
> * KIP-405 : Kafka Tiered Storage
> * KIP-868 : KRaft Metadata Transactions
> * KIP-875: First-class offsets support in Kafka Connect
> * KIP-898: Modernize Connect plugin discovery
> * KIP-938: Add more metrics for measuring KRaft performance
> * KIP-902: Upgrade Zookeeper to 3.8.1
> * KIP-917: Additional custom metadata for remote log segment
>
> Release notes for the 3.6.0 release:
> https://home.apache.org/~satishd/kafka-3.6.0-rc1/RELEASE_NOTES.html
>
> *** Please download, test and vote by Saturday, September 23, 8am PT
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> https://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> https://home.apache.org/~satishd/kafka-3.6.0-rc1/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>
> * Javadoc:
> https://home.apache.org/~satishd/kafka-3.6.0-rc1/javadoc/
>
> * Tag to be voted upon (off 3.6 branch) is the 3.6.0 tag:
> https://github.com/apache/kafka/releases/tag/3.6.0-rc1
>
> * Documentation:
> https://kafka.apache.org/36/documentation.html
>
> * Protocol:
> https://kafka.apache.org/36/protocol.html
>
> * Successful Jenkins builds for the 3.6 branch:
> There are a few runs of unit/integration tests. You can see the latest at 
> https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.6/. We will continue 
> running a few more iterations.
> System tests:
> We will send an update once we have the results.
>
> Thanks,
> Satish.
>
> --
> You received this message because you are subscribed to the Google Groups 
> "kafka-clients" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to kafka-clients+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/kafka-clients/CAM-aUZ%3DuJ-SKeVFtBZwBjhLHKw4CbxF_ws%2BvQqaymGHFsC%2Bmdg%40mail.gmail.com.

Re: [kafka-clients] [VOTE] 3.6.0 RC1

[Greg Harris]

Hi all,

I verified the functionality of KIP-898 and the recent fix for
KAFKA-15473 with the following steps:

1. I started a 3.5.1 broker, and a 3.5.1 worker with most (>400)
publicly available plugins installed
2. I captured the output of /connector-plugins
3. I upgraded the worker to 3.6.0-rc1
4. I captured the output of /connector-plugins with various settings
of plugin.discovery
5. I ran the migration script to add manifests to my plugins
6. I captured the output of /connector-plugins with various settings
of plugin.discovery
7. I downgraded the worker to 3.5.1
8. I diffed the output of /connector-plugins across the different
cases and observed the expected changes.
a. When plugins are migrated for 3.6.0, all modes produce identical results.
b. When plugins are not migrated for 3.6.0, only_scan and
hybrid_warn produce identical results, hybrid_fail crashes, and
service_load is missing plugins
c. When upgrading from 3.5.1 I see that plugins with invalid
constructors are hidden, AK plugins now have versions, multi-interface
plugins now show each interface type, and plugins using AppInfoParser
change versions.
d. The startup logs now include descriptive errors for invalid
plugins that otherwise would have been thrown at runtime
d. The fix for KAFKA-15473 prevents duplicates
e. The output for 3.5.1 after downgrading is identical to before.

+1 (non-binding)

Thanks Satish for running the release!

On Wed, Sep 20, 2023 at 8:36 AM Divij Vaidya  wrote:
>
> Hey Satish
>
> My comments about documentation misses from RC0 vote thread [1] are
> still not addressed (such as missing metric documentation, formatting
> problems etc). Could you please mention why we shouldn't consider them
> as blockers to make RC1 as the final release?
>
> [1] https://lists.apache.org/thread/cokoxzd0jtgjtrlxoq7kkzmvpm75381t
>
> On Wed, Sep 20, 2023 at 4:53 PM Satish Duggana  
> wrote:
> >
> > Hello Kafka users, developers and client-developers,
> >
> > This is the second candidate for the release of Apache Kafka 3.6.0. Some of 
> > the major features include:
> >
> > * KIP-405 : Kafka Tiered Storage
> > * KIP-868 : KRaft Metadata Transactions
> > * KIP-875: First-class offsets support in Kafka Connect
> > * KIP-898: Modernize Connect plugin discovery
> > * KIP-938: Add more metrics for measuring KRaft performance
> > * KIP-902: Upgrade Zookeeper to 3.8.1
> > * KIP-917: Additional custom metadata for remote log segment
> >
> > Release notes for the 3.6.0 release:
> > https://home.apache.org/~satishd/kafka-3.6.0-rc1/RELEASE_NOTES.html
> >
> > *** Please download, test and vote by Saturday, September 23, 8am PT
> >
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > https://kafka.apache.org/KEYS
> >
> > * Release artifacts to be voted upon (source and binary):
> > https://home.apache.org/~satishd/kafka-3.6.0-rc1/
> >
> > * Maven artifacts to be voted upon:
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >
> > * Javadoc:
> > https://home.apache.org/~satishd/kafka-3.6.0-rc1/javadoc/
> >
> > * Tag to be voted upon (off 3.6 branch) is the 3.6.0 tag:
> > https://github.com/apache/kafka/releases/tag/3.6.0-rc1
> >
> > * Documentation:
> > https://kafka.apache.org/36/documentation.html
> >
> > * Protocol:
> > https://kafka.apache.org/36/protocol.html
> >
> > * Successful Jenkins builds for the 3.6 branch:
> > There are a few runs of unit/integration tests. You can see the latest at 
> > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.6/. We will continue 
> > running a few more iterations.
> > System tests:
> > We will send an update once we have the results.
> >
> > Thanks,
> > Satish.
> >
> > --
> > You received this message because you are subscribed to the Google Groups 
> > "kafka-clients" group.
> > To unsubscribe from this group and stop receiving emails from it, send an 
> > email to kafka-clients+unsubscr...@googlegroups.com.
> > To view this discussion on the web visit 
> > https://groups.google.com/d/msgid/kafka-clients/CAM-aUZ%3DuJ-SKeVFtBZwBjhLHKw4CbxF_ws%2BvQqaymGHFsC%2Bmdg%40mail.gmail.com.

Re: [kafka-clients] [VOTE] 3.6.0 RC1

[Gary Russell]

This is a blocker for me [1].

https://issues.apache.org/jira/browse/KAFKA-15482


From: Greg Harris 
Sent: Wednesday, September 20, 2023 12:45 PM
To: users@kafka.apache.org 
Cc: dev ; kafka-clie...@googlegroups.com 

Subject: Re: [kafka-clients] [VOTE] 3.6.0 RC1

!! External Email

Hi all,

I verified the functionality of KIP-898 and the recent fix for
KAFKA-15473 with the following steps:

1. I started a 3.5.1 broker, and a 3.5.1 worker with most (>400)
publicly available plugins installed
2. I captured the output of /connector-plugins
3. I upgraded the worker to 3.6.0-rc1
4. I captured the output of /connector-plugins with various settings
of plugin.discovery
5. I ran the migration script to add manifests to my plugins
6. I captured the output of /connector-plugins with various settings
of plugin.discovery
7. I downgraded the worker to 3.5.1
8. I diffed the output of /connector-plugins across the different
cases and observed the expected changes.
a. When plugins are migrated for 3.6.0, all modes produce identical results.
b. When plugins are not migrated for 3.6.0, only_scan and
hybrid_warn produce identical results, hybrid_fail crashes, and
service_load is missing plugins
c. When upgrading from 3.5.1 I see that plugins with invalid
constructors are hidden, AK plugins now have versions, multi-interface
plugins now show each interface type, and plugins using AppInfoParser
change versions.
d. The startup logs now include descriptive errors for invalid
plugins that otherwise would have been thrown at runtime
d. The fix for KAFKA-15473 prevents duplicates
e. The output for 3.5.1 after downgrading is identical to before.

+1 (non-binding)

Thanks Satish for running the release!

On Wed, Sep 20, 2023 at 8:36 AM Divij Vaidya  wrote:
>
> Hey Satish
>
> My comments about documentation misses from RC0 vote thread [1] are
> still not addressed (such as missing metric documentation, formatting
> problems etc). Could you please mention why we shouldn't consider them
> as blockers to make RC1 as the final release?
>
> [1] 
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Fcokoxzd0jtgjtrlxoq7kkzmvpm75381t&data=05%7C01%7Cgrussell%40vmware.com%7C573bc3210fd64ebf23e308dbb9f93c12%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C638308252383167831%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=RTJ3LZ8uWUsTsxwMMO5B6izLcG1r%2F9il%2F98kvBkjz2Y%3D&reserved=0<https://lists.apache.org/thread/cokoxzd0jtgjtrlxoq7kkzmvpm75381t>
>
> On Wed, Sep 20, 2023 at 4:53 PM Satish Duggana  
> wrote:
> >
> > Hello Kafka users, developers and client-developers,
> >
> > This is the second candidate for the release of Apache Kafka 3.6.0. Some of 
> > the major features include:
> >
> > * KIP-405 : Kafka Tiered Storage
> > * KIP-868 : KRaft Metadata Transactions
> > * KIP-875: First-class offsets support in Kafka Connect
> > * KIP-898: Modernize Connect plugin discovery
> > * KIP-938: Add more metrics for measuring KRaft performance
> > * KIP-902: Upgrade Zookeeper to 3.8.1
> > * KIP-917: Additional custom metadata for remote log segment
> >
> > Release notes for the 3.6.0 release:
> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhome.apache.org%2F~satishd%2Fkafka-3.6.0-rc1%2FRELEASE_NOTES.html&data=05%7C01%7Cgrussell%40vmware.com%7C573bc3210fd64ebf23e308dbb9f93c12%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C638308252383324078%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=qx8EKVV0QTKwU5GnqrJNSppZYr2UnKeZ3H%2BPGmuj3QM%3D&reserved=0<https://home.apache.org/~satishd/kafka-3.6.0-rc1/RELEASE_NOTES.html>
> >
> > *** Please download, test and vote by Saturday, September 23, 8am PT
> >
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fkafka.apache.org%2FKEYS&data=05%7C01%7Cgrussell%40vmware.com%7C573bc3210fd64ebf23e308dbb9f93c12%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C638308252383324078%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=vs2aCLRV%2Fha5yLjvuExhj8CQ45rwM0M%2BZHSqOVQU2i4%3D&reserved=0<https://kafka.apache.org/KEYS>
> >
> > * Release artifacts to be voted upon (source and binary):
> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhome.apache.org%2F~satishd%2Fkafka-3.6.0-rc1%2F&data=05%7C01%7Cgrussell%40vmware.com%7C573bc3210fd64ebf23e308dbb9f93c12%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C638308252383324078%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiM

Re: [kafka-clients] [VOTE] 3.6.0 RC1

[Gary Russell]

I have lowered the priority because I found a work around, but it's still bad 
practice for a public class to depend on a class that is not published anywhere.

This is similar to [2], which has been open for months.

[1]: https://issues.apache.org/jira/browse/KAFKA-15482
[2]: https://issues.apache.org/jira/browse/KAFKA-15092

From: Gary Russell 
Sent: Wednesday, September 20, 2023 3:36 PM
To: users@kafka.apache.org ; dev 
Subject: Re: [kafka-clients] [VOTE] 3.6.0 RC1

This is a blocker for me [1].

https://issues.apache.org/jira/browse/KAFKA-15482


From: Greg Harris 
Sent: Wednesday, September 20, 2023 12:45 PM
To: users@kafka.apache.org 
Cc: dev ; kafka-clie...@googlegroups.com 

Subject: Re: [kafka-clients] [VOTE] 3.6.0 RC1

!! External Email

Hi all,

I verified the functionality of KIP-898 and the recent fix for
KAFKA-15473 with the following steps:

1. I started a 3.5.1 broker, and a 3.5.1 worker with most (>400)
publicly available plugins installed
2. I captured the output of /connector-plugins
3. I upgraded the worker to 3.6.0-rc1
4. I captured the output of /connector-plugins with various settings
of plugin.discovery
5. I ran the migration script to add manifests to my plugins
6. I captured the output of /connector-plugins with various settings
of plugin.discovery
7. I downgraded the worker to 3.5.1
8. I diffed the output of /connector-plugins across the different
cases and observed the expected changes.
a. When plugins are migrated for 3.6.0, all modes produce identical results.
b. When plugins are not migrated for 3.6.0, only_scan and
hybrid_warn produce identical results, hybrid_fail crashes, and
service_load is missing plugins
c. When upgrading from 3.5.1 I see that plugins with invalid
constructors are hidden, AK plugins now have versions, multi-interface
plugins now show each interface type, and plugins using AppInfoParser
change versions.
d. The startup logs now include descriptive errors for invalid
plugins that otherwise would have been thrown at runtime
d. The fix for KAFKA-15473 prevents duplicates
e. The output for 3.5.1 after downgrading is identical to before.

+1 (non-binding)

Thanks Satish for running the release!

On Wed, Sep 20, 2023 at 8:36 AM Divij Vaidya  wrote:
>
> Hey Satish
>
> My comments about documentation misses from RC0 vote thread [1] are
> still not addressed (such as missing metric documentation, formatting
> problems etc). Could you please mention why we shouldn't consider them
> as blockers to make RC1 as the final release?
>
> [1] 
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Fcokoxzd0jtgjtrlxoq7kkzmvpm75381t&data=05%7C01%7Cgrussell%40vmware.com%7C573bc3210fd64ebf23e308dbb9f93c12%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C638308252383167831%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=RTJ3LZ8uWUsTsxwMMO5B6izLcG1r%2F9il%2F98kvBkjz2Y%3D&reserved=0<https://lists.apache.org/thread/cokoxzd0jtgjtrlxoq7kkzmvpm75381t>
>
> On Wed, Sep 20, 2023 at 4:53 PM Satish Duggana  
> wrote:
> >
> > Hello Kafka users, developers and client-developers,
> >
> > This is the second candidate for the release of Apache Kafka 3.6.0. Some of 
> > the major features include:
> >
> > * KIP-405 : Kafka Tiered Storage
> > * KIP-868 : KRaft Metadata Transactions
> > * KIP-875: First-class offsets support in Kafka Connect
> > * KIP-898: Modernize Connect plugin discovery
> > * KIP-938: Add more metrics for measuring KRaft performance
> > * KIP-902: Upgrade Zookeeper to 3.8.1
> > * KIP-917: Additional custom metadata for remote log segment
> >
> > Release notes for the 3.6.0 release:
> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhome.apache.org%2F~satishd%2Fkafka-3.6.0-rc1%2FRELEASE_NOTES.html&data=05%7C01%7Cgrussell%40vmware.com%7C573bc3210fd64ebf23e308dbb9f93c12%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C638308252383324078%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=qx8EKVV0QTKwU5GnqrJNSppZYr2UnKeZ3H%2BPGmuj3QM%3D&reserved=0<https://home.apache.org/~satishd/kafka-3.6.0-rc1/RELEASE_NOTES.html>
> >
> > *** Please download, test and vote by Saturday, September 23, 8am PT
> >
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fkafka.apache.org%2FKEYS&data=05%7C01%7Cgrussell%40vmware.com%7C573bc3210fd64ebf23e308dbb9f93c12%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C638308252383324078%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sd

Re: [kafka-clients] [VOTE] 3.6.0 RC1

[Gary Russell]

My apologies; both classes are in the server-common...-test jar.

Both issues can be closed.

From: Gary Russell 
Sent: Wednesday, September 20, 2023 3:56 PM
To: users@kafka.apache.org ; dev 
Subject: Re: [kafka-clients] [VOTE] 3.6.0 RC1

I have lowered the priority because I found a work around, but it's still bad 
practice for a public class to depend on a class that is not published anywhere.

This is similar to [2], which has been open for months.

[1]: https://issues.apache.org/jira/browse/KAFKA-15482
[2]: https://issues.apache.org/jira/browse/KAFKA-15092

From: Gary Russell 
Sent: Wednesday, September 20, 2023 3:36 PM
To: users@kafka.apache.org ; dev 
Subject: Re: [kafka-clients] [VOTE] 3.6.0 RC1

This is a blocker for me [1].

https://issues.apache.org/jira/browse/KAFKA-15482


From: Greg Harris 
Sent: Wednesday, September 20, 2023 12:45 PM
To: users@kafka.apache.org 
Cc: dev ; kafka-clie...@googlegroups.com 

Subject: Re: [kafka-clients] [VOTE] 3.6.0 RC1

!! External Email

Hi all,

I verified the functionality of KIP-898 and the recent fix for
KAFKA-15473 with the following steps:

1. I started a 3.5.1 broker, and a 3.5.1 worker with most (>400)
publicly available plugins installed
2. I captured the output of /connector-plugins
3. I upgraded the worker to 3.6.0-rc1
4. I captured the output of /connector-plugins with various settings
of plugin.discovery
5. I ran the migration script to add manifests to my plugins
6. I captured the output of /connector-plugins with various settings
of plugin.discovery
7. I downgraded the worker to 3.5.1
8. I diffed the output of /connector-plugins across the different
cases and observed the expected changes.
a. When plugins are migrated for 3.6.0, all modes produce identical results.
b. When plugins are not migrated for 3.6.0, only_scan and
hybrid_warn produce identical results, hybrid_fail crashes, and
service_load is missing plugins
c. When upgrading from 3.5.1 I see that plugins with invalid
constructors are hidden, AK plugins now have versions, multi-interface
plugins now show each interface type, and plugins using AppInfoParser
change versions.
d. The startup logs now include descriptive errors for invalid
plugins that otherwise would have been thrown at runtime
d. The fix for KAFKA-15473 prevents duplicates
e. The output for 3.5.1 after downgrading is identical to before.

+1 (non-binding)

Thanks Satish for running the release!

On Wed, Sep 20, 2023 at 8:36 AM Divij Vaidya  wrote:
>
> Hey Satish
>
> My comments about documentation misses from RC0 vote thread [1] are
> still not addressed (such as missing metric documentation, formatting
> problems etc). Could you please mention why we shouldn't consider them
> as blockers to make RC1 as the final release?
>
> [1] 
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Fcokoxzd0jtgjtrlxoq7kkzmvpm75381t&data=05%7C01%7Cgrussell%40vmware.com%7C573bc3210fd64ebf23e308dbb9f93c12%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C638308252383167831%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=RTJ3LZ8uWUsTsxwMMO5B6izLcG1r%2F9il%2F98kvBkjz2Y%3D&reserved=0<https://lists.apache.org/thread/cokoxzd0jtgjtrlxoq7kkzmvpm75381t>
>
> On Wed, Sep 20, 2023 at 4:53 PM Satish Duggana  
> wrote:
> >
> > Hello Kafka users, developers and client-developers,
> >
> > This is the second candidate for the release of Apache Kafka 3.6.0. Some of 
> > the major features include:
> >
> > * KIP-405 : Kafka Tiered Storage
> > * KIP-868 : KRaft Metadata Transactions
> > * KIP-875: First-class offsets support in Kafka Connect
> > * KIP-898: Modernize Connect plugin discovery
> > * KIP-938: Add more metrics for measuring KRaft performance
> > * KIP-902: Upgrade Zookeeper to 3.8.1
> > * KIP-917: Additional custom metadata for remote log segment
> >
> > Release notes for the 3.6.0 release:
> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhome.apache.org%2F~satishd%2Fkafka-3.6.0-rc1%2FRELEASE_NOTES.html&data=05%7C01%7Cgrussell%40vmware.com%7C573bc3210fd64ebf23e308dbb9f93c12%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C638308252383324078%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=qx8EKVV0QTKwU5GnqrJNSppZYr2UnKeZ3H%2BPGmuj3QM%3D&reserved=0<https://home.apache.org/~satishd/kafka-3.6.0-rc1/RELEASE_NOTES.html>
> >
> > *** Please download, test and vote by Saturday, September 23, 8am PT
> >
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > https://nam04.sa

Re: [kafka-clients] [VOTE] 3.6.0 RC1

[Satish Duggana]

Hi Divij,
Thanks for calling out documentation issues. Some of them are being
addressed in the existing documentation PRs. We will keep addressing the
remaining documentation issues by making respective changes in kafka-site
repo for 3.6.0 release.

I think documentation is important but I do not consider documentation
issues as release blockers. I want others to review and test the release
candidate sooner to finalize the RC for 3.6.0 release.

~Satish.

On Wed, 20 Sept 2023 at 21:04, Divij Vaidya  wrote:

> Hey Satish
>
> My comments about documentation misses from RC0 vote thread [1] are
> still not addressed (such as missing metric documentation, formatting
> problems etc). Could you please mention why we shouldn't consider them
> as blockers to make RC1 as the final release?
>
> [1] https://lists.apache.org/thread/cokoxzd0jtgjtrlxoq7kkzmvpm75381t
>
> On Wed, Sep 20, 2023 at 4:53 PM Satish Duggana 
> wrote:
> >
> > Hello Kafka users, developers and client-developers,
> >
> > This is the second candidate for the release of Apache Kafka 3.6.0. Some
> of the major features include:
> >
> > * KIP-405 : Kafka Tiered Storage
> > * KIP-868 : KRaft Metadata Transactions
> > * KIP-875: First-class offsets support in Kafka Connect
> > * KIP-898: Modernize Connect plugin discovery
> > * KIP-938: Add more metrics for measuring KRaft performance
> > * KIP-902: Upgrade Zookeeper to 3.8.1
> > * KIP-917: Additional custom metadata for remote log segment
> >
> > Release notes for the 3.6.0 release:
> > https://home.apache.org/~satishd/kafka-3.6.0-rc1/RELEASE_NOTES.html
> >
> > *** Please download, test and vote by Saturday, September 23, 8am PT
> >
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > https://kafka.apache.org/KEYS
> >
> > * Release artifacts to be voted upon (source and binary):
> > https://home.apache.org/~satishd/kafka-3.6.0-rc1/
> >
> > * Maven artifacts to be voted upon:
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >
> > * Javadoc:
> > https://home.apache.org/~satishd/kafka-3.6.0-rc1/javadoc/
> >
> > * Tag to be voted upon (off 3.6 branch) is the 3.6.0 tag:
> > https://github.com/apache/kafka/releases/tag/3.6.0-rc1
> >
> > * Documentation:
> > https://kafka.apache.org/36/documentation.html
> >
> > * Protocol:
> > https://kafka.apache.org/36/protocol.html
> >
> > * Successful Jenkins builds for the 3.6 branch:
> > There are a few runs of unit/integration tests. You can see the latest
> at https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.6/. We will
> continue running a few more iterations.
> > System tests:
> > We will send an update once we have the results.
> >
> > Thanks,
> > Satish.
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups "kafka-clients" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to kafka-clients+unsubscr...@googlegroups.com.
> > To view this discussion on the web visit
> https://groups.google.com/d/msgid/kafka-clients/CAM-aUZ%3DuJ-SKeVFtBZwBjhLHKw4CbxF_ws%2BvQqaymGHFsC%2Bmdg%40mail.gmail.com
> .
>

Re: [kafka-clients] [VOTE] 3.6.0 RC1

[Satish Duggana]

Thanks Greg for verifying the release including the earlier
blocker(KAFKA-15473) verification.

~Satish.

On Wed, 20 Sept 2023 at 22:30, Greg Harris 
wrote:

> Hi all,
>
> I verified the functionality of KIP-898 and the recent fix for
> KAFKA-15473 with the following steps:
>
> 1. I started a 3.5.1 broker, and a 3.5.1 worker with most (>400)
> publicly available plugins installed
> 2. I captured the output of /connector-plugins
> 3. I upgraded the worker to 3.6.0-rc1
> 4. I captured the output of /connector-plugins with various settings
> of plugin.discovery
> 5. I ran the migration script to add manifests to my plugins
> 6. I captured the output of /connector-plugins with various settings
> of plugin.discovery
> 7. I downgraded the worker to 3.5.1
> 8. I diffed the output of /connector-plugins across the different
> cases and observed the expected changes.
> a. When plugins are migrated for 3.6.0, all modes produce identical
> results.
> b. When plugins are not migrated for 3.6.0, only_scan and
> hybrid_warn produce identical results, hybrid_fail crashes, and
> service_load is missing plugins
> c. When upgrading from 3.5.1 I see that plugins with invalid
> constructors are hidden, AK plugins now have versions, multi-interface
> plugins now show each interface type, and plugins using AppInfoParser
> change versions.
> d. The startup logs now include descriptive errors for invalid
> plugins that otherwise would have been thrown at runtime
> d. The fix for KAFKA-15473 prevents duplicates
> e. The output for 3.5.1 after downgrading is identical to before.
>
> +1 (non-binding)
>
> Thanks Satish for running the release!
>
> On Wed, Sep 20, 2023 at 8:36 AM Divij Vaidya  wrote:
> >
> > Hey Satish
> >
> > My comments about documentation misses from RC0 vote thread [1] are
> > still not addressed (such as missing metric documentation, formatting
> > problems etc). Could you please mention why we shouldn't consider them
> > as blockers to make RC1 as the final release?
> >
> > [1] https://lists.apache.org/thread/cokoxzd0jtgjtrlxoq7kkzmvpm75381t
> >
> > On Wed, Sep 20, 2023 at 4:53 PM Satish Duggana 
> wrote:
> > >
> > > Hello Kafka users, developers and client-developers,
> > >
> > > This is the second candidate for the release of Apache Kafka 3.6.0.
> Some of the major features include:
> > >
> > > * KIP-405 : Kafka Tiered Storage
> > > * KIP-868 : KRaft Metadata Transactions
> > > * KIP-875: First-class offsets support in Kafka Connect
> > > * KIP-898: Modernize Connect plugin discovery
> > > * KIP-938: Add more metrics for measuring KRaft performance
> > > * KIP-902: Upgrade Zookeeper to 3.8.1
> > > * KIP-917: Additional custom metadata for remote log segment
> > >
> > > Release notes for the 3.6.0 release:
> > > https://home.apache.org/~satishd/kafka-3.6.0-rc1/RELEASE_NOTES.html
> > >
> > > *** Please download, test and vote by Saturday, September 23, 8am PT
> > >
> > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > https://kafka.apache.org/KEYS
> > >
> > > * Release artifacts to be voted upon (source and binary):
> > > https://home.apache.org/~satishd/kafka-3.6.0-rc1/
> > >
> > > * Maven artifacts to be voted upon:
> > > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > >
> > > * Javadoc:
> > > https://home.apache.org/~satishd/kafka-3.6.0-rc1/javadoc/
> > >
> > > * Tag to be voted upon (off 3.6 branch) is the 3.6.0 tag:
> > > https://github.com/apache/kafka/releases/tag/3.6.0-rc1
> > >
> > > * Documentation:
> > > https://kafka.apache.org/36/documentation.html
> > >
> > > * Protocol:
> > > https://kafka.apache.org/36/protocol.html
> > >
> > > * Successful Jenkins builds for the 3.6 branch:
> > > There are a few runs of unit/integration tests. You can see the latest
> at https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.6/. We will
> continue running a few more iterations.
> > > System tests:
> > > We will send an update once we have the results.
> > >
> > > Thanks,
> > > Satish.
> > >
> > > --
> > > You received this message because you are subscribed to the Google
> Groups "kafka-clients" group.
> > > To unsubscribe from this group and stop receiving emails from it, send
> an email to kafka-clients+unsubscr...@googlegroups.com.
> > > To view this discussion on the web visit
> https://groups.google.com/d/msgid/kafka-clients/CAM-aUZ%3DuJ-SKeVFtBZwBjhLHKw4CbxF_ws%2BvQqaymGHFsC%2Bmdg%40mail.gmail.com
> .
>

Re: [kafka-clients] [VOTE] 3.6.0 RC1

[Josep Prat]

Hi Satish,

I ran the following validation steps:
- Built from source with Java 11 and Scala 2.13
- Verified Signatures and hashes of the artifacts generated
- Navigated through Javadoc including links to JDK classes
- Run the unit tests
- Run integration tests
- Run the quickstart in KRaft and Zookeeper mode


I +1 this release (non-binding)

Thanks for your efforts!

On Thu, Sep 21, 2023 at 2:59 AM Satish Duggana 
wrote:

> Thanks Greg for verifying the release including the earlier
> blocker(KAFKA-15473) verification.
>
> ~Satish.
>
> On Wed, 20 Sept 2023 at 22:30, Greg Harris 
> wrote:
>
> > Hi all,
> >
> > I verified the functionality of KIP-898 and the recent fix for
> > KAFKA-15473 with the following steps:
> >
> > 1. I started a 3.5.1 broker, and a 3.5.1 worker with most (>400)
> > publicly available plugins installed
> > 2. I captured the output of /connector-plugins
> > 3. I upgraded the worker to 3.6.0-rc1
> > 4. I captured the output of /connector-plugins with various settings
> > of plugin.discovery
> > 5. I ran the migration script to add manifests to my plugins
> > 6. I captured the output of /connector-plugins with various settings
> > of plugin.discovery
> > 7. I downgraded the worker to 3.5.1
> > 8. I diffed the output of /connector-plugins across the different
> > cases and observed the expected changes.
> > a. When plugins are migrated for 3.6.0, all modes produce identical
> > results.
> > b. When plugins are not migrated for 3.6.0, only_scan and
> > hybrid_warn produce identical results, hybrid_fail crashes, and
> > service_load is missing plugins
> > c. When upgrading from 3.5.1 I see that plugins with invalid
> > constructors are hidden, AK plugins now have versions, multi-interface
> > plugins now show each interface type, and plugins using AppInfoParser
> > change versions.
> > d. The startup logs now include descriptive errors for invalid
> > plugins that otherwise would have been thrown at runtime
> > d. The fix for KAFKA-15473 prevents duplicates
> > e. The output for 3.5.1 after downgrading is identical to before.
> >
> > +1 (non-binding)
> >
> > Thanks Satish for running the release!
> >
> > On Wed, Sep 20, 2023 at 8:36 AM Divij Vaidya  wrote:
> > >
> > > Hey Satish
> > >
> > > My comments about documentation misses from RC0 vote thread [1] are
> > > still not addressed (such as missing metric documentation, formatting
> > > problems etc). Could you please mention why we shouldn't consider them
> > > as blockers to make RC1 as the final release?
> > >
> > > [1] https://lists.apache.org/thread/cokoxzd0jtgjtrlxoq7kkzmvpm75381t
> > >
> > > On Wed, Sep 20, 2023 at 4:53 PM Satish Duggana <
> satish.dugg...@gmail.com>
> > wrote:
> > > >
> > > > Hello Kafka users, developers and client-developers,
> > > >
> > > > This is the second candidate for the release of Apache Kafka 3.6.0.
> > Some of the major features include:
> > > >
> > > > * KIP-405 : Kafka Tiered Storage
> > > > * KIP-868 : KRaft Metadata Transactions
> > > > * KIP-875: First-class offsets support in Kafka Connect
> > > > * KIP-898: Modernize Connect plugin discovery
> > > > * KIP-938: Add more metrics for measuring KRaft performance
> > > > * KIP-902: Upgrade Zookeeper to 3.8.1
> > > > * KIP-917: Additional custom metadata for remote log segment
> > > >
> > > > Release notes for the 3.6.0 release:
> > > > https://home.apache.org/~satishd/kafka-3.6.0-rc1/RELEASE_NOTES.html
> > > >
> > > > *** Please download, test and vote by Saturday, September 23, 8am PT
> > > >
> > > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > > https://kafka.apache.org/KEYS
> > > >
> > > > * Release artifacts to be voted upon (source and binary):
> > > > https://home.apache.org/~satishd/kafka-3.6.0-rc1/
> > > >
> > > > * Maven artifacts to be voted upon:
> > > >
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > >
> > > > * Javadoc:
> > > > https://home.apache.org/~satishd/kafka-3.6.0-rc1/javadoc/
> > > >
> > > > * Tag to be voted upon (off 3.6 branch) is the 3.6.0 tag:
> > > > https://github.com/apache/kafka/releases/tag/3.6.0-rc1
> > > >
> > > > * Documentation:
> > > > https://kafka.apache.org/36/documentation.html
> > > >
> > > > * Protocol:
> > > > https://kafka.apache.org/36/protocol.html
> > > >
> > > > * Successful Jenkins builds for the 3.6 branch:
> > > > There are a few runs of unit/integration tests. You can see the
> latest
> > at https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.6/. We will
> > continue running a few more iterations.
> > > > System tests:
> > > > We will send an update once we have the results.
> > > >
> > > > Thanks,
> > > > Satish.
> > > >
> > > > --
> > > > You received this message because you are subscribed to the Google
> > Groups "kafka-clients" group.
> > > > To unsubscribe from this group and stop receiving emails from it,
> send
> > an email to kafka-clients+unsubscr...@googlegroups.com.
> > > > To view this discussion on t

Re: [kafka-clients] [VOTE] 3.6.0 RC1

[Luke Chen]

Hi Satish,

I found the current KRaft implementation will have "split brain" issue when
network partition happens, which will cause inconsistent metadata returned
from the controller.
Filed KAFKA-15489  for
this issue, and PR  is ready
for review.

Even though this is not a regression issue (this has already existed since
the 1st release of KRaft feature), I think this is an important issue since
KRaft is announced production ready.
Not sure what other people's thoughts are.

Thank you.
Luke

On Thu, Sep 21, 2023 at 6:33 PM Josep Prat 
wrote:

> Hi Satish,
>
> I ran the following validation steps:
> - Built from source with Java 11 and Scala 2.13
> - Verified Signatures and hashes of the artifacts generated
> - Navigated through Javadoc including links to JDK classes
> - Run the unit tests
> - Run integration tests
> - Run the quickstart in KRaft and Zookeeper mode
>
>
> I +1 this release (non-binding)
>
> Thanks for your efforts!
>
> On Thu, Sep 21, 2023 at 2:59 AM Satish Duggana 
> wrote:
>
> > Thanks Greg for verifying the release including the earlier
> > blocker(KAFKA-15473) verification.
> >
> > ~Satish.
> >
> > On Wed, 20 Sept 2023 at 22:30, Greg Harris  >
> > wrote:
> >
> > > Hi all,
> > >
> > > I verified the functionality of KIP-898 and the recent fix for
> > > KAFKA-15473 with the following steps:
> > >
> > > 1. I started a 3.5.1 broker, and a 3.5.1 worker with most (>400)
> > > publicly available plugins installed
> > > 2. I captured the output of /connector-plugins
> > > 3. I upgraded the worker to 3.6.0-rc1
> > > 4. I captured the output of /connector-plugins with various settings
> > > of plugin.discovery
> > > 5. I ran the migration script to add manifests to my plugins
> > > 6. I captured the output of /connector-plugins with various settings
> > > of plugin.discovery
> > > 7. I downgraded the worker to 3.5.1
> > > 8. I diffed the output of /connector-plugins across the different
> > > cases and observed the expected changes.
> > > a. When plugins are migrated for 3.6.0, all modes produce identical
> > > results.
> > > b. When plugins are not migrated for 3.6.0, only_scan and
> > > hybrid_warn produce identical results, hybrid_fail crashes, and
> > > service_load is missing plugins
> > > c. When upgrading from 3.5.1 I see that plugins with invalid
> > > constructors are hidden, AK plugins now have versions, multi-interface
> > > plugins now show each interface type, and plugins using AppInfoParser
> > > change versions.
> > > d. The startup logs now include descriptive errors for invalid
> > > plugins that otherwise would have been thrown at runtime
> > > d. The fix for KAFKA-15473 prevents duplicates
> > > e. The output for 3.5.1 after downgrading is identical to before.
> > >
> > > +1 (non-binding)
> > >
> > > Thanks Satish for running the release!
> > >
> > > On Wed, Sep 20, 2023 at 8:36 AM Divij Vaidya 
> wrote:
> > > >
> > > > Hey Satish
> > > >
> > > > My comments about documentation misses from RC0 vote thread [1] are
> > > > still not addressed (such as missing metric documentation, formatting
> > > > problems etc). Could you please mention why we shouldn't consider
> them
> > > > as blockers to make RC1 as the final release?
> > > >
> > > > [1] https://lists.apache.org/thread/cokoxzd0jtgjtrlxoq7kkzmvpm75381t
> > > >
> > > > On Wed, Sep 20, 2023 at 4:53 PM Satish Duggana <
> > satish.dugg...@gmail.com>
> > > wrote:
> > > > >
> > > > > Hello Kafka users, developers and client-developers,
> > > > >
> > > > > This is the second candidate for the release of Apache Kafka 3.6.0.
> > > Some of the major features include:
> > > > >
> > > > > * KIP-405 : Kafka Tiered Storage
> > > > > * KIP-868 : KRaft Metadata Transactions
> > > > > * KIP-875: First-class offsets support in Kafka Connect
> > > > > * KIP-898: Modernize Connect plugin discovery
> > > > > * KIP-938: Add more metrics for measuring KRaft performance
> > > > > * KIP-902: Upgrade Zookeeper to 3.8.1
> > > > > * KIP-917: Additional custom metadata for remote log segment
> > > > >
> > > > > Release notes for the 3.6.0 release:
> > > > >
> https://home.apache.org/~satishd/kafka-3.6.0-rc1/RELEASE_NOTES.html
> > > > >
> > > > > *** Please download, test and vote by Saturday, September 23, 8am
> PT
> > > > >
> > > > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > > > https://kafka.apache.org/KEYS
> > > > >
> > > > > * Release artifacts to be voted upon (source and binary):
> > > > > https://home.apache.org/~satishd/kafka-3.6.0-rc1/
> > > > >
> > > > > * Maven artifacts to be voted upon:
> > > > >
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > > >
> > > > > * Javadoc:
> > > > > https://home.apache.org/~satishd/kafka-3.6.0-rc1/javadoc/
> > > > >
> > > > > * Tag to be voted upon (off 3.6 branch) is the 3.6.0 tag:
> > > > > https://github.com/apache/kafka/releases/

Re: [kafka-clients] [VOTE] 3.6.0 RC1

[Satish Duggana]

Thanks Luke for reporting KRaft issue[1].

I am not sure whether it is a release blocker for 3.6.0. Need input
from other KRaft experts also to finalize the decision. Even if we
adopt a fix, do not we need to bake it for some time before it is
pushed to production to avoid any regressions as this change is in the
critical paths?

1. https://issues.apache.org/jira/browse/KAFKA-15489

Thanks,
Satish.

On Sat, 23 Sept 2023 at 03:08, Luke Chen  wrote:
>
> Hi Satish,
>
> I found the current KRaft implementation will have "split brain" issue when
> network partition happens, which will cause inconsistent metadata returned
> from the controller.
> Filed KAFKA-15489  for
> this issue, and PR  is ready
> for review.
>
> Even though this is not a regression issue (this has already existed since
> the 1st release of KRaft feature), I think this is an important issue since
> KRaft is announced production ready.
> Not sure what other people's thoughts are.
>
> Thank you.
> Luke
>
> On Thu, Sep 21, 2023 at 6:33 PM Josep Prat 
> wrote:
>
> > Hi Satish,
> >
> > I ran the following validation steps:
> > - Built from source with Java 11 and Scala 2.13
> > - Verified Signatures and hashes of the artifacts generated
> > - Navigated through Javadoc including links to JDK classes
> > - Run the unit tests
> > - Run integration tests
> > - Run the quickstart in KRaft and Zookeeper mode
> >
> >
> > I +1 this release (non-binding)
> >
> > Thanks for your efforts!
> >
> > On Thu, Sep 21, 2023 at 2:59 AM Satish Duggana 
> > wrote:
> >
> > > Thanks Greg for verifying the release including the earlier
> > > blocker(KAFKA-15473) verification.
> > >
> > > ~Satish.
> > >
> > > On Wed, 20 Sept 2023 at 22:30, Greg Harris  > >
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I verified the functionality of KIP-898 and the recent fix for
> > > > KAFKA-15473 with the following steps:
> > > >
> > > > 1. I started a 3.5.1 broker, and a 3.5.1 worker with most (>400)
> > > > publicly available plugins installed
> > > > 2. I captured the output of /connector-plugins
> > > > 3. I upgraded the worker to 3.6.0-rc1
> > > > 4. I captured the output of /connector-plugins with various settings
> > > > of plugin.discovery
> > > > 5. I ran the migration script to add manifests to my plugins
> > > > 6. I captured the output of /connector-plugins with various settings
> > > > of plugin.discovery
> > > > 7. I downgraded the worker to 3.5.1
> > > > 8. I diffed the output of /connector-plugins across the different
> > > > cases and observed the expected changes.
> > > > a. When plugins are migrated for 3.6.0, all modes produce identical
> > > > results.
> > > > b. When plugins are not migrated for 3.6.0, only_scan and
> > > > hybrid_warn produce identical results, hybrid_fail crashes, and
> > > > service_load is missing plugins
> > > > c. When upgrading from 3.5.1 I see that plugins with invalid
> > > > constructors are hidden, AK plugins now have versions, multi-interface
> > > > plugins now show each interface type, and plugins using AppInfoParser
> > > > change versions.
> > > > d. The startup logs now include descriptive errors for invalid
> > > > plugins that otherwise would have been thrown at runtime
> > > > d. The fix for KAFKA-15473 prevents duplicates
> > > > e. The output for 3.5.1 after downgrading is identical to before.
> > > >
> > > > +1 (non-binding)
> > > >
> > > > Thanks Satish for running the release!
> > > >
> > > > On Wed, Sep 20, 2023 at 8:36 AM Divij Vaidya 
> > wrote:
> > > > >
> > > > > Hey Satish
> > > > >
> > > > > My comments about documentation misses from RC0 vote thread [1] are
> > > > > still not addressed (such as missing metric documentation, formatting
> > > > > problems etc). Could you please mention why we shouldn't consider
> > them
> > > > > as blockers to make RC1 as the final release?
> > > > >
> > > > > [1] https://lists.apache.org/thread/cokoxzd0jtgjtrlxoq7kkzmvpm75381t
> > > > >
> > > > > On Wed, Sep 20, 2023 at 4:53 PM Satish Duggana <
> > > satish.dugg...@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > Hello Kafka users, developers and client-developers,
> > > > > >
> > > > > > This is the second candidate for the release of Apache Kafka 3.6.0.
> > > > Some of the major features include:
> > > > > >
> > > > > > * KIP-405 : Kafka Tiered Storage
> > > > > > * KIP-868 : KRaft Metadata Transactions
> > > > > > * KIP-875: First-class offsets support in Kafka Connect
> > > > > > * KIP-898: Modernize Connect plugin discovery
> > > > > > * KIP-938: Add more metrics for measuring KRaft performance
> > > > > > * KIP-902: Upgrade Zookeeper to 3.8.1
> > > > > > * KIP-917: Additional custom metadata for remote log segment
> > > > > >
> > > > > > Release notes for the 3.6.0 release:
> > > > > >
> > https://home.apache.org/~satishd/kafka-3.6.0-rc1/RELEASE_NOTES.html
> > > > > >
> > > > > > *** Please download, 

Re: [kafka-clients] [VOTE] 3.6.0 RC1

[Ismael Juma]

Given that this is not a regression and there have been no reports for over
a year, I think it's ok for this to land in 3.6.1.

Ismael

On Sat, Sep 23, 2023 at 9:32 AM Satish Duggana 
wrote:

> Thanks Luke for reporting KRaft issue[1].
>
> I am not sure whether it is a release blocker for 3.6.0. Need input
> from other KRaft experts also to finalize the decision. Even if we
> adopt a fix, do not we need to bake it for some time before it is
> pushed to production to avoid any regressions as this change is in the
> critical paths?
>
> 1. https://issues.apache.org/jira/browse/KAFKA-15489
>
> Thanks,
> Satish.
>
> On Sat, 23 Sept 2023 at 03:08, Luke Chen  wrote:
> >
> > Hi Satish,
> >
> > I found the current KRaft implementation will have "split brain" issue
> when
> > network partition happens, which will cause inconsistent metadata
> returned
> > from the controller.
> > Filed KAFKA-15489 
> for
> > this issue, and PR  is ready
> > for review.
> >
> > Even though this is not a regression issue (this has already existed
> since
> > the 1st release of KRaft feature), I think this is an important issue
> since
> > KRaft is announced production ready.
> > Not sure what other people's thoughts are.
> >
> > Thank you.
> > Luke
> >
> > On Thu, Sep 21, 2023 at 6:33 PM Josep Prat 
> > wrote:
> >
> > > Hi Satish,
> > >
> > > I ran the following validation steps:
> > > - Built from source with Java 11 and Scala 2.13
> > > - Verified Signatures and hashes of the artifacts generated
> > > - Navigated through Javadoc including links to JDK classes
> > > - Run the unit tests
> > > - Run integration tests
> > > - Run the quickstart in KRaft and Zookeeper mode
> > >
> > >
> > > I +1 this release (non-binding)
> > >
> > > Thanks for your efforts!
> > >
> > > On Thu, Sep 21, 2023 at 2:59 AM Satish Duggana <
> satish.dugg...@gmail.com>
> > > wrote:
> > >
> > > > Thanks Greg for verifying the release including the earlier
> > > > blocker(KAFKA-15473) verification.
> > > >
> > > > ~Satish.
> > > >
> > > > On Wed, 20 Sept 2023 at 22:30, Greg Harris
>  > > >
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I verified the functionality of KIP-898 and the recent fix for
> > > > > KAFKA-15473 with the following steps:
> > > > >
> > > > > 1. I started a 3.5.1 broker, and a 3.5.1 worker with most (>400)
> > > > > publicly available plugins installed
> > > > > 2. I captured the output of /connector-plugins
> > > > > 3. I upgraded the worker to 3.6.0-rc1
> > > > > 4. I captured the output of /connector-plugins with various
> settings
> > > > > of plugin.discovery
> > > > > 5. I ran the migration script to add manifests to my plugins
> > > > > 6. I captured the output of /connector-plugins with various
> settings
> > > > > of plugin.discovery
> > > > > 7. I downgraded the worker to 3.5.1
> > > > > 8. I diffed the output of /connector-plugins across the different
> > > > > cases and observed the expected changes.
> > > > > a. When plugins are migrated for 3.6.0, all modes produce
> identical
> > > > > results.
> > > > > b. When plugins are not migrated for 3.6.0, only_scan and
> > > > > hybrid_warn produce identical results, hybrid_fail crashes, and
> > > > > service_load is missing plugins
> > > > > c. When upgrading from 3.5.1 I see that plugins with invalid
> > > > > constructors are hidden, AK plugins now have versions,
> multi-interface
> > > > > plugins now show each interface type, and plugins using
> AppInfoParser
> > > > > change versions.
> > > > > d. The startup logs now include descriptive errors for invalid
> > > > > plugins that otherwise would have been thrown at runtime
> > > > > d. The fix for KAFKA-15473 prevents duplicates
> > > > > e. The output for 3.5.1 after downgrading is identical to
> before.
> > > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > Thanks Satish for running the release!
> > > > >
> > > > > On Wed, Sep 20, 2023 at 8:36 AM Divij Vaidya 
> > > wrote:
> > > > > >
> > > > > > Hey Satish
> > > > > >
> > > > > > My comments about documentation misses from RC0 vote thread [1]
> are
> > > > > > still not addressed (such as missing metric documentation,
> formatting
> > > > > > problems etc). Could you please mention why we shouldn't consider
> > > them
> > > > > > as blockers to make RC1 as the final release?
> > > > > >
> > > > > > [1]
> https://lists.apache.org/thread/cokoxzd0jtgjtrlxoq7kkzmvpm75381t
> > > > > >
> > > > > > On Wed, Sep 20, 2023 at 4:53 PM Satish Duggana <
> > > > satish.dugg...@gmail.com>
> > > > > wrote:
> > > > > > >
> > > > > > > Hello Kafka users, developers and client-developers,
> > > > > > >
> > > > > > > This is the second candidate for the release of Apache Kafka
> 3.6.0.
> > > > > Some of the major features include:
> > > > > > >
> > > > > > > * KIP-405 : Kafka Tiered Storage
> > > > > > > * KIP-868 : KRaft Metadata Transactions
> > > > > > > * KIP-

Re: [kafka-clients] [VOTE] 3.6.0 RC1

[Luke Chen]

Hi Satish,

I verified with:
1. Ran quick start in KRaft for scala 2.12 artifact
2. Making sure the checksum are correct
3. Browsing release notes, documents, javadocs, protocols.

I filed KAFKA-15491 for
log output improvement while testing stream application.
It won't be blocker in v3.6.0.

For KAFKA-15489 , I'm
fine if we decide to fix it in v3.6.1/v3.7.0.

+1 (binding) from me.

Thank you.
Luke

On Sun, Sep 24, 2023 at 3:38 AM Ismael Juma  wrote:

> Given that this is not a regression and there have been no reports for over
> a year, I think it's ok for this to land in 3.6.1.
>
> Ismael
>
> On Sat, Sep 23, 2023 at 9:32 AM Satish Duggana 
> wrote:
>
> > Thanks Luke for reporting KRaft issue[1].
> >
> > I am not sure whether it is a release blocker for 3.6.0. Need input
> > from other KRaft experts also to finalize the decision. Even if we
> > adopt a fix, do not we need to bake it for some time before it is
> > pushed to production to avoid any regressions as this change is in the
> > critical paths?
> >
> > 1. https://issues.apache.org/jira/browse/KAFKA-15489
> >
> > Thanks,
> > Satish.
> >
> > On Sat, 23 Sept 2023 at 03:08, Luke Chen  wrote:
> > >
> > > Hi Satish,
> > >
> > > I found the current KRaft implementation will have "split brain" issue
> > when
> > > network partition happens, which will cause inconsistent metadata
> > returned
> > > from the controller.
> > > Filed KAFKA-15489 
> > for
> > > this issue, and PR  is
> ready
> > > for review.
> > >
> > > Even though this is not a regression issue (this has already existed
> > since
> > > the 1st release of KRaft feature), I think this is an important issue
> > since
> > > KRaft is announced production ready.
> > > Not sure what other people's thoughts are.
> > >
> > > Thank you.
> > > Luke
> > >
> > > On Thu, Sep 21, 2023 at 6:33 PM Josep Prat  >
> > > wrote:
> > >
> > > > Hi Satish,
> > > >
> > > > I ran the following validation steps:
> > > > - Built from source with Java 11 and Scala 2.13
> > > > - Verified Signatures and hashes of the artifacts generated
> > > > - Navigated through Javadoc including links to JDK classes
> > > > - Run the unit tests
> > > > - Run integration tests
> > > > - Run the quickstart in KRaft and Zookeeper mode
> > > >
> > > >
> > > > I +1 this release (non-binding)
> > > >
> > > > Thanks for your efforts!
> > > >
> > > > On Thu, Sep 21, 2023 at 2:59 AM Satish Duggana <
> > satish.dugg...@gmail.com>
> > > > wrote:
> > > >
> > > > > Thanks Greg for verifying the release including the earlier
> > > > > blocker(KAFKA-15473) verification.
> > > > >
> > > > > ~Satish.
> > > > >
> > > > > On Wed, 20 Sept 2023 at 22:30, Greg Harris
> >  > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I verified the functionality of KIP-898 and the recent fix for
> > > > > > KAFKA-15473 with the following steps:
> > > > > >
> > > > > > 1. I started a 3.5.1 broker, and a 3.5.1 worker with most (>400)
> > > > > > publicly available plugins installed
> > > > > > 2. I captured the output of /connector-plugins
> > > > > > 3. I upgraded the worker to 3.6.0-rc1
> > > > > > 4. I captured the output of /connector-plugins with various
> > settings
> > > > > > of plugin.discovery
> > > > > > 5. I ran the migration script to add manifests to my plugins
> > > > > > 6. I captured the output of /connector-plugins with various
> > settings
> > > > > > of plugin.discovery
> > > > > > 7. I downgraded the worker to 3.5.1
> > > > > > 8. I diffed the output of /connector-plugins across the different
> > > > > > cases and observed the expected changes.
> > > > > > a. When plugins are migrated for 3.6.0, all modes produce
> > identical
> > > > > > results.
> > > > > > b. When plugins are not migrated for 3.6.0, only_scan and
> > > > > > hybrid_warn produce identical results, hybrid_fail crashes, and
> > > > > > service_load is missing plugins
> > > > > > c. When upgrading from 3.5.1 I see that plugins with invalid
> > > > > > constructors are hidden, AK plugins now have versions,
> > multi-interface
> > > > > > plugins now show each interface type, and plugins using
> > AppInfoParser
> > > > > > change versions.
> > > > > > d. The startup logs now include descriptive errors for
> invalid
> > > > > > plugins that otherwise would have been thrown at runtime
> > > > > > d. The fix for KAFKA-15473 prevents duplicates
> > > > > > e. The output for 3.5.1 after downgrading is identical to
> > before.
> > > > > >
> > > > > > +1 (non-binding)
> > > > > >
> > > > > > Thanks Satish for running the release!
> > > > > >
> > > > > > On Wed, Sep 20, 2023 at 8:36 AM Divij Vaidya 
> > > > wrote:
> > > > > > >
> > > > > > > Hey Satish
> > > > > > >
> > > > > > > My comments about documentation misses from RC0 vote thread [1]
> > are

Re: [kafka-clients] [VOTE] 3.6.0 RC1

[Federico Valeri]

Hi Satish, I did the following to verify the release:

- Verified signature and checksum
- Built from source with Java 17 and Scala 2.13
- Ran all unit and integration tests
- Spot checked release notes and documentation
- Ran a custom client using staging artifacts on a 3-nodes cluster
- Tested tiered storage with one of the available RSM implementations

+1 (non binding)

Thanks
Fede


On Sun, Sep 24, 2023 at 8:49 AM Luke Chen  wrote:
>
> Hi Satish,
>
> I verified with:
> 1. Ran quick start in KRaft for scala 2.12 artifact
> 2. Making sure the checksum are correct
> 3. Browsing release notes, documents, javadocs, protocols.
>
> I filed KAFKA-15491 for
> log output improvement while testing stream application.
> It won't be blocker in v3.6.0.
>
> For KAFKA-15489 , I'm
> fine if we decide to fix it in v3.6.1/v3.7.0.
>
> +1 (binding) from me.
>
> Thank you.
> Luke
>
> On Sun, Sep 24, 2023 at 3:38 AM Ismael Juma  wrote:
>
> > Given that this is not a regression and there have been no reports for over
> > a year, I think it's ok for this to land in 3.6.1.
> >
> > Ismael
> >
> > On Sat, Sep 23, 2023 at 9:32 AM Satish Duggana 
> > wrote:
> >
> > > Thanks Luke for reporting KRaft issue[1].
> > >
> > > I am not sure whether it is a release blocker for 3.6.0. Need input
> > > from other KRaft experts also to finalize the decision. Even if we
> > > adopt a fix, do not we need to bake it for some time before it is
> > > pushed to production to avoid any regressions as this change is in the
> > > critical paths?
> > >
> > > 1. https://issues.apache.org/jira/browse/KAFKA-15489
> > >
> > > Thanks,
> > > Satish.
> > >
> > > On Sat, 23 Sept 2023 at 03:08, Luke Chen  wrote:
> > > >
> > > > Hi Satish,
> > > >
> > > > I found the current KRaft implementation will have "split brain" issue
> > > when
> > > > network partition happens, which will cause inconsistent metadata
> > > returned
> > > > from the controller.
> > > > Filed KAFKA-15489 
> > > for
> > > > this issue, and PR  is
> > ready
> > > > for review.
> > > >
> > > > Even though this is not a regression issue (this has already existed
> > > since
> > > > the 1st release of KRaft feature), I think this is an important issue
> > > since
> > > > KRaft is announced production ready.
> > > > Not sure what other people's thoughts are.
> > > >
> > > > Thank you.
> > > > Luke
> > > >
> > > > On Thu, Sep 21, 2023 at 6:33 PM Josep Prat  > >
> > > > wrote:
> > > >
> > > > > Hi Satish,
> > > > >
> > > > > I ran the following validation steps:
> > > > > - Built from source with Java 11 and Scala 2.13
> > > > > - Verified Signatures and hashes of the artifacts generated
> > > > > - Navigated through Javadoc including links to JDK classes
> > > > > - Run the unit tests
> > > > > - Run integration tests
> > > > > - Run the quickstart in KRaft and Zookeeper mode
> > > > >
> > > > >
> > > > > I +1 this release (non-binding)
> > > > >
> > > > > Thanks for your efforts!
> > > > >
> > > > > On Thu, Sep 21, 2023 at 2:59 AM Satish Duggana <
> > > satish.dugg...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Thanks Greg for verifying the release including the earlier
> > > > > > blocker(KAFKA-15473) verification.
> > > > > >
> > > > > > ~Satish.
> > > > > >
> > > > > > On Wed, 20 Sept 2023 at 22:30, Greg Harris
> > >  > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > I verified the functionality of KIP-898 and the recent fix for
> > > > > > > KAFKA-15473 with the following steps:
> > > > > > >
> > > > > > > 1. I started a 3.5.1 broker, and a 3.5.1 worker with most (>400)
> > > > > > > publicly available plugins installed
> > > > > > > 2. I captured the output of /connector-plugins
> > > > > > > 3. I upgraded the worker to 3.6.0-rc1
> > > > > > > 4. I captured the output of /connector-plugins with various
> > > settings
> > > > > > > of plugin.discovery
> > > > > > > 5. I ran the migration script to add manifests to my plugins
> > > > > > > 6. I captured the output of /connector-plugins with various
> > > settings
> > > > > > > of plugin.discovery
> > > > > > > 7. I downgraded the worker to 3.5.1
> > > > > > > 8. I diffed the output of /connector-plugins across the different
> > > > > > > cases and observed the expected changes.
> > > > > > > a. When plugins are migrated for 3.6.0, all modes produce
> > > identical
> > > > > > > results.
> > > > > > > b. When plugins are not migrated for 3.6.0, only_scan and
> > > > > > > hybrid_warn produce identical results, hybrid_fail crashes, and
> > > > > > > service_load is missing plugins
> > > > > > > c. When upgrading from 3.5.1 I see that plugins with invalid
> > > > > > > constructors are hidden, AK plugins now have versions,
> > > multi-interface
> > > > > > > plugins now show each interface typ

Re: [kafka-clients] [VOTE] 3.6.0 RC1

[José Armando García Sancio]

On Sat, Sep 23, 2023 at 3:08 AM Luke Chen  wrote:
>
> Hi Satish,
>
> I found the current KRaft implementation will have "split brain" issue when
> network partition happens, which will cause inconsistent metadata returned
> from the controller.
> Filed KAFKA-15489  for
> this issue, and PR  is ready
> for review.
>
> Even though this is not a regression issue (this has already existed since
> the 1st release of KRaft feature), I think this is an important issue since
> KRaft is announced production ready.
> Not sure what other people's thoughts are.

Thanks for the report and PR Luke. This looks related to this issue:
https://issues.apache.org/jira/browse/KAFKA-13621

Do you agree? We can move our conversation to those issues but I also
agree that I don't think this issue should be a release blocker.

Thanks!
-José

Re: [kafka-clients] [VOTE] 3.6.0 RC1

[Luke Chen]

Hi Jose,

Sounds good to me.
Let's have further discussion in JIRA/PR, and target to v3.6.1/v3.7.0.

Thanks.
Luke

On Tue, Sep 26, 2023 at 1:35 AM José Armando García Sancio
 wrote:

> On Sat, Sep 23, 2023 at 3:08 AM Luke Chen  wrote:
> >
> > Hi Satish,
> >
> > I found the current KRaft implementation will have "split brain" issue
> when
> > network partition happens, which will cause inconsistent metadata
> returned
> > from the controller.
> > Filed KAFKA-15489 
> for
> > this issue, and PR  is ready
> > for review.
> >
> > Even though this is not a regression issue (this has already existed
> since
> > the 1st release of KRaft feature), I think this is an important issue
> since
> > KRaft is announced production ready.
> > Not sure what other people's thoughts are.
>
> Thanks for the report and PR Luke. This looks related to this issue:
> https://issues.apache.org/jira/browse/KAFKA-13621
>
> Do you agree? We can move our conversation to those issues but I also
> agree that I don't think this issue should be a release blocker.
>
> Thanks!
> -José
>