Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2217

[Apache Jenkins Server]

See 

Re: [kafka-clients] [VOTE] 3.6.0 RC1

[Satish Duggana]

Thanks Greg for verifying the release including the earlier
blocker(KAFKA-15473) verification.

~Satish.

On Wed, 20 Sept 2023 at 22:30, Greg Harris 
wrote:

> Hi all,
>
> I verified the functionality of KIP-898 and the recent fix for
> KAFKA-15473 with the following steps:
>
> 1. I started a 3.5.1 broker, and a 3.5.1 worker with most (>400)
> publicly available plugins installed
> 2. I captured the output of /connector-plugins
> 3. I upgraded the worker to 3.6.0-rc1
> 4. I captured the output of /connector-plugins with various settings
> of plugin.discovery
> 5. I ran the migration script to add manifests to my plugins
> 6. I captured the output of /connector-plugins with various settings
> of plugin.discovery
> 7. I downgraded the worker to 3.5.1
> 8. I diffed the output of /connector-plugins across the different
> cases and observed the expected changes.
> a. When plugins are migrated for 3.6.0, all modes produce identical
> results.
> b. When plugins are not migrated for 3.6.0, only_scan and
> hybrid_warn produce identical results, hybrid_fail crashes, and
> service_load is missing plugins
> c. When upgrading from 3.5.1 I see that plugins with invalid
> constructors are hidden, AK plugins now have versions, multi-interface
> plugins now show each interface type, and plugins using AppInfoParser
> change versions.
> d. The startup logs now include descriptive errors for invalid
> plugins that otherwise would have been thrown at runtime
> d. The fix for KAFKA-15473 prevents duplicates
> e. The output for 3.5.1 after downgrading is identical to before.
>
> +1 (non-binding)
>
> Thanks Satish for running the release!
>
> On Wed, Sep 20, 2023 at 8:36 AM Divij Vaidya  wrote:
> >
> > Hey Satish
> >
> > My comments about documentation misses from RC0 vote thread [1] are
> > still not addressed (such as missing metric documentation, formatting
> > problems etc). Could you please mention why we shouldn't consider them
> > as blockers to make RC1 as the final release?
> >
> > [1] https://lists.apache.org/thread/cokoxzd0jtgjtrlxoq7kkzmvpm75381t
> >
> > On Wed, Sep 20, 2023 at 4:53 PM Satish Duggana 
> wrote:
> > >
> > > Hello Kafka users, developers and client-developers,
> > >
> > > This is the second candidate for the release of Apache Kafka 3.6.0.
> Some of the major features include:
> > >
> > > * KIP-405 : Kafka Tiered Storage
> > > * KIP-868 : KRaft Metadata Transactions
> > > * KIP-875: First-class offsets support in Kafka Connect
> > > * KIP-898: Modernize Connect plugin discovery
> > > * KIP-938: Add more metrics for measuring KRaft performance
> > > * KIP-902: Upgrade Zookeeper to 3.8.1
> > > * KIP-917: Additional custom metadata for remote log segment
> > >
> > > Release notes for the 3.6.0 release:
> > > https://home.apache.org/~satishd/kafka-3.6.0-rc1/RELEASE_NOTES.html
> > >
> > > *** Please download, test and vote by Saturday, September 23, 8am PT
> > >
> > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > https://kafka.apache.org/KEYS
> > >
> > > * Release artifacts to be voted upon (source and binary):
> > > https://home.apache.org/~satishd/kafka-3.6.0-rc1/
> > >
> > > * Maven artifacts to be voted upon:
> > > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > >
> > > * Javadoc:
> > > https://home.apache.org/~satishd/kafka-3.6.0-rc1/javadoc/
> > >
> > > * Tag to be voted upon (off 3.6 branch) is the 3.6.0 tag:
> > > https://github.com/apache/kafka/releases/tag/3.6.0-rc1
> > >
> > > * Documentation:
> > > https://kafka.apache.org/36/documentation.html
> > >
> > > * Protocol:
> > > https://kafka.apache.org/36/protocol.html
> > >
> > > * Successful Jenkins builds for the 3.6 branch:
> > > There are a few runs of unit/integration tests. You can see the latest
> at https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.6/. We will
> continue running a few more iterations.
> > > System tests:
> > > We will send an update once we have the results.
> > >
> > > Thanks,
> > > Satish.
> > >
> > > --
> > > You received this message because you are subscribed to the Google
> Groups "kafka-clients" group.
> > > To unsubscribe from this group and stop receiving emails from it, send
> an email to kafka-clients+unsubscr...@googlegroups.com.
> > > To view this discussion on the web visit
> https://groups.google.com/d/msgid/kafka-clients/CAM-aUZ%3DuJ-SKeVFtBZwBjhLHKw4CbxF_ws%2BvQqaymGHFsC%2Bmdg%40mail.gmail.com
> .
>

Re: [kafka-clients] [VOTE] 3.6.0 RC1

[Satish Duggana]

Hi Divij,
Thanks for calling out documentation issues. Some of them are being
addressed in the existing documentation PRs. We will keep addressing the
remaining documentation issues by making respective changes in kafka-site
repo for 3.6.0 release.

I think documentation is important but I do not consider documentation
issues as release blockers. I want others to review and test the release
candidate sooner to finalize the RC for 3.6.0 release.

~Satish.

On Wed, 20 Sept 2023 at 21:04, Divij Vaidya  wrote:

> Hey Satish
>
> My comments about documentation misses from RC0 vote thread [1] are
> still not addressed (such as missing metric documentation, formatting
> problems etc). Could you please mention why we shouldn't consider them
> as blockers to make RC1 as the final release?
>
> [1] https://lists.apache.org/thread/cokoxzd0jtgjtrlxoq7kkzmvpm75381t
>
> On Wed, Sep 20, 2023 at 4:53 PM Satish Duggana 
> wrote:
> >
> > Hello Kafka users, developers and client-developers,
> >
> > This is the second candidate for the release of Apache Kafka 3.6.0. Some
> of the major features include:
> >
> > * KIP-405 : Kafka Tiered Storage
> > * KIP-868 : KRaft Metadata Transactions
> > * KIP-875: First-class offsets support in Kafka Connect
> > * KIP-898: Modernize Connect plugin discovery
> > * KIP-938: Add more metrics for measuring KRaft performance
> > * KIP-902: Upgrade Zookeeper to 3.8.1
> > * KIP-917: Additional custom metadata for remote log segment
> >
> > Release notes for the 3.6.0 release:
> > https://home.apache.org/~satishd/kafka-3.6.0-rc1/RELEASE_NOTES.html
> >
> > *** Please download, test and vote by Saturday, September 23, 8am PT
> >
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > https://kafka.apache.org/KEYS
> >
> > * Release artifacts to be voted upon (source and binary):
> > https://home.apache.org/~satishd/kafka-3.6.0-rc1/
> >
> > * Maven artifacts to be voted upon:
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >
> > * Javadoc:
> > https://home.apache.org/~satishd/kafka-3.6.0-rc1/javadoc/
> >
> > * Tag to be voted upon (off 3.6 branch) is the 3.6.0 tag:
> > https://github.com/apache/kafka/releases/tag/3.6.0-rc1
> >
> > * Documentation:
> > https://kafka.apache.org/36/documentation.html
> >
> > * Protocol:
> > https://kafka.apache.org/36/protocol.html
> >
> > * Successful Jenkins builds for the 3.6 branch:
> > There are a few runs of unit/integration tests. You can see the latest
> at https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.6/. We will
> continue running a few more iterations.
> > System tests:
> > We will send an update once we have the results.
> >
> > Thanks,
> > Satish.
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups "kafka-clients" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to kafka-clients+unsubscr...@googlegroups.com.
> > To view this discussion on the web visit
> https://groups.google.com/d/msgid/kafka-clients/CAM-aUZ%3DuJ-SKeVFtBZwBjhLHKw4CbxF_ws%2BvQqaymGHFsC%2Bmdg%40mail.gmail.com
> .
>

[jira] [Created] (KAFKA-15483) Update metrics documentation for the new metrics implemented as part of KIP-938

[Satish Duggana (Jira)]

Satish Duggana created KAFKA-15483:
--

 Summary: Update metrics documentation for the new metrics 
implemented as part of KIP-938
 Key: KAFKA-15483
 URL: https://issues.apache.org/jira/browse/KAFKA-15483
 Project: Kafka
  Issue Type: Task
  Components: docs, documentation
Reporter: Satish Duggana






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE]KIP-966: Eligible Leader Replicas

[Justine Olshan]

Thanks Calvin.
I think this will be very helpful going forward to minimize data loss.

+1 from me (binding)

Justine

On Wed, Sep 20, 2023 at 3:42 PM Calvin Liu 
wrote:

> Hi all,
> I'd like to call for a vote on KIP-966 which includes a series of
> enhancements to the current ISR model.
>
>- Introduce the new HWM advancement requirement which enables the system
>to have more potentially data-safe replicas.
>- Introduce Eligible Leader Replicas(ELR) to represent the above
>data-safe replicas.
>- Introduce Unclean Recovery process which will deterministically choose
>the best replica during an unclean leader election.
>
>
> KIP:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-966%3A+Eligible+Leader+Replicas
>
> Discussion thread:
> https://lists.apache.org/thread/gpbpx9kpd7c62dm962h6kww0ghgznb38
>

[jira] [Resolved] (KAFKA-15092) KafkaClusterTestKit in test jar depends on MockFaultHandler

[Satish Duggana (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Duggana resolved KAFKA-15092.

Resolution: Invalid

> KafkaClusterTestKit in test jar depends on MockFaultHandler
> ---
>
> Key: KAFKA-15092
> URL: https://issues.apache.org/jira/browse/KAFKA-15092
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 3.5.0
>Reporter: Gary Russell
>Priority: Major
>
> {noformat}
> java.lang.NoClassDefFoundError: org/apache/kafka/server/fault/MockFaultHandler
>   at 
> kafka.testkit.KafkaClusterTestKit$SimpleFaultHandlerFactory.(KafkaClusterTestKit.java:119)
>   at 
> kafka.testkit.KafkaClusterTestKit$Builder.(KafkaClusterTestKit.java:143)
> {noformat}
> MockFaultHandler is missing from the test jar.
> This PR https://github.com/apache/kafka/pull/13375/files seems to work around 
> it by adding the {code}server-common sourcesets.test.output{code} to the 
> class path.
> The class needs to be available for third parties to create an embedded KRaft 
> broker.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15482) kafka.utils.TestUtils Depends on MockTime Which is Not in Any Jar

[Satish Duggana (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Duggana resolved KAFKA-15482.

Resolution: Invalid

> kafka.utils.TestUtils Depends on MockTime Which is Not in Any Jar
> -
>
> Key: KAFKA-15482
> URL: https://issues.apache.org/jira/browse/KAFKA-15482
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 3.6.0
>Reporter: Gary Russell
>Priority: Major
>
> Commit 
> 7eea2a3908fdcee1627c18827e6dcb5ed0089fdd 
> Moved it to server-commons, but it is not included in the jar.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[VOTE]KIP-966: Eligible Leader Replicas

[Calvin Liu]

Hi all,
I'd like to call for a vote on KIP-966 which includes a series of
enhancements to the current ISR model.

   - Introduce the new HWM advancement requirement which enables the system
   to have more potentially data-safe replicas.
   - Introduce Eligible Leader Replicas(ELR) to represent the above
   data-safe replicas.
   - Introduce Unclean Recovery process which will deterministically choose
   the best replica during an unclean leader election.


KIP:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-966%3A+Eligible+Leader+Replicas

Discussion thread:
https://lists.apache.org/thread/gpbpx9kpd7c62dm962h6kww0ghgznb38

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2216

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-979: Allow independently stop KRaft controllers or brokers

[Hailey Ni]

Hi Ron,

Thank you very much for the comment. I think it makes sense to me that we
provide an even more specific way to kill individual controllers/brokers.
I have one question: does the command line for ZooKeeper cluster provide
such a way to kill individual controllers/brokers?

Thanks,
Hailey

On Tue, Sep 19, 2023 at 11:01 AM Ron Dagostino  wrote:

> Thanks for the KIP, Hailey.  It will be nice to provide some
> fine-grained control for when people running the broker and controller
> this way want to stop just one of them.
>
> One thing that occurs to me is that in a development environment
> someone might want to run multiple controllers and multiple brokers
> all on the same box, and in that case they might want to actually stop
> just one controller or just one broker instead of all of them.  So I'm
> wondering if maybe instead of supporting kafka-server-stop
> [--process.roles ] we might want to instead support
> kafka-server-stop [--required-config ].  If someone wanted
> to stop any/all controllers and not touch the broker(s) they could
> still achieve that by invoking kafka-server-stop --required-config
> process.roles=controller.  But if they did want to stop a particular
> controller they could then also achieve that via kafka-server-stop
> --required-config node.id=1 (for example).  What do you think?
>
> Ron
>
> On Thu, Sep 14, 2023 at 5:56 PM Hailey Ni 
> wrote:
> >
> > Hi all,
> >
> > I would like to start the discussion about *KIP-979: Allow independently
> > stop KRaft controllers or brokers* <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-979%3A+Allow+independently+stop+KRaft+controllers+or+brokers
> > >
> > It proposes adding an optional field "--process.roles " in the
> > script to allow users to independently stop either KRaft broker processes
> > or controller processes. While in the past, all processes were killed
> using
> > a single script.
> > Please let me know if you have any questions or comments. Much
> appreciated.
> >
> > Thanks & Regards,
> > Hailey
>

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.6 #58

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 305359 lines...]

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldAddTask() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldNotAssignAnyLockedTask() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldNotAssignAnyLockedTask() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldRemoveTask() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldRemoveTask() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldNotRemoveAssignedTask() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldNotRemoveAssignedTask() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldAssignTaskThatCanBeProcessed() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldAssignTaskThatCanBeProcessed() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldNotRemoveUnlockedTask() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldNotRemoveUnlockedTask() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldReturnAndClearExceptionsOnDrainExceptions() 
STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldReturnAndClearExceptionsOnDrainExceptions() 
PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldUnassignTask() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldUnassignTask() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > 
shouldNotAssignTasksForProcessingIfProcessingDisabled() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > 
shouldNotAssignTasksForProcessingIfProcessingDisabled() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldNotSetUncaughtExceptionsForUnassignedTasks() 
STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldNotSetUncaughtExceptionsForUnassignedTasks() 
PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldNotAssignLockedTask() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldNotAssignLockedTask() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldUnassignLockingTask() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldUnassignLockingTask() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldAssignTasksThatCanBeStreamTimePunctuated() 
STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldAssignTasksThatCanBeStreamTimePunctuated() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
RocksDBTimeOrderedKeyValueBytesStoreTest > shouldCreateEmptyWriteBatches() 
STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
RocksDBTimeOrderedKeyValueBytesStoreTest > shouldCreateEmptyWriteBatches() 
PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
RocksDBTimeOrderedKeyValueBytesStoreTest > shouldCreateWriteBatches() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
RocksDBTimeOrderedKeyValueBytesStoreTest > shouldCreateWriteBatches() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
RocksDBMetricsRecorderTest > 
shouldThrowIfDbToAddWasAlreadyAddedForOtherSegment() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
RocksDBMetricsRecorderTest > 
shouldThrowIfDbToAddWasAlreadyAddedForOtherSegment() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
RocksDBMetricsRecorderTest > 
shouldAddItselfToRecordingTriggerWhenFirstValueProvidersAreAddedAfterLastValueProvidersWereRemoved()
 STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
RocksDBMetricsRecorderTest > 
shouldAddItselfToRecordingTriggerWhenFirstValueProvidersAreAddedAfterLastValueProvidersWereRemoved()
 PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
RocksDBMetricsRecorderTest > shouldThrowIfValueProvidersToRemoveNotFound() 
STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
RocksDBMetricsRecorderTest > shouldThrowIfValueProvidersToRemoveNotFound() 
PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 

Re: [kafka-clients] [VOTE] 3.6.0 RC1

[Gary Russell]

My apologies; both classes are in the server-common...-test jar.

Both issues can be closed.

From: Gary Russell 
Sent: Wednesday, September 20, 2023 3:56 PM
To: us...@kafka.apache.org ; dev 
Subject: Re: [kafka-clients] [VOTE] 3.6.0 RC1

I have lowered the priority because I found a work around, but it's still bad 
practice for a public class to depend on a class that is not published anywhere.

This is similar to [2], which has been open for months.

[1]: https://issues.apache.org/jira/browse/KAFKA-15482
[2]: https://issues.apache.org/jira/browse/KAFKA-15092

From: Gary Russell 
Sent: Wednesday, September 20, 2023 3:36 PM
To: us...@kafka.apache.org ; dev 
Subject: Re: [kafka-clients] [VOTE] 3.6.0 RC1

This is a blocker for me [1].

https://issues.apache.org/jira/browse/KAFKA-15482


From: Greg Harris 
Sent: Wednesday, September 20, 2023 12:45 PM
To: us...@kafka.apache.org 
Cc: dev ; kafka-clie...@googlegroups.com 

Subject: Re: [kafka-clients] [VOTE] 3.6.0 RC1

!! External Email

Hi all,

I verified the functionality of KIP-898 and the recent fix for
KAFKA-15473 with the following steps:

1. I started a 3.5.1 broker, and a 3.5.1 worker with most (>400)
publicly available plugins installed
2. I captured the output of /connector-plugins
3. I upgraded the worker to 3.6.0-rc1
4. I captured the output of /connector-plugins with various settings
of plugin.discovery
5. I ran the migration script to add manifests to my plugins
6. I captured the output of /connector-plugins with various settings
of plugin.discovery
7. I downgraded the worker to 3.5.1
8. I diffed the output of /connector-plugins across the different
cases and observed the expected changes.
a. When plugins are migrated for 3.6.0, all modes produce identical results.
b. When plugins are not migrated for 3.6.0, only_scan and
hybrid_warn produce identical results, hybrid_fail crashes, and
service_load is missing plugins
c. When upgrading from 3.5.1 I see that plugins with invalid
constructors are hidden, AK plugins now have versions, multi-interface
plugins now show each interface type, and plugins using AppInfoParser
change versions.
d. The startup logs now include descriptive errors for invalid
plugins that otherwise would have been thrown at runtime
d. The fix for KAFKA-15473 prevents duplicates
e. The output for 3.5.1 after downgrading is identical to before.

+1 (non-binding)

Thanks Satish for running the release!

On Wed, Sep 20, 2023 at 8:36 AM Divij Vaidya  wrote:
>
> Hey Satish
>
> My comments about documentation misses from RC0 vote thread [1] are
> still not addressed (such as missing metric documentation, formatting
> problems etc). Could you please mention why we shouldn't consider them
> as blockers to make RC1 as the final release?
>
> [1] 
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Fcokoxzd0jtgjtrlxoq7kkzmvpm75381t=05%7C01%7Cgrussell%40vmware.com%7C573bc3210fd64ebf23e308dbb9f93c12%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C638308252383167831%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=RTJ3LZ8uWUsTsxwMMO5B6izLcG1r%2F9il%2F98kvBkjz2Y%3D=0
>
> On Wed, Sep 20, 2023 at 4:53 PM Satish Duggana  
> wrote:
> >
> > Hello Kafka users, developers and client-developers,
> >
> > This is the second candidate for the release of Apache Kafka 3.6.0. Some of 
> > the major features include:
> >
> > * KIP-405 : Kafka Tiered Storage
> > * KIP-868 : KRaft Metadata Transactions
> > * KIP-875: First-class offsets support in Kafka Connect
> > * KIP-898: Modernize Connect plugin discovery
> > * KIP-938: Add more metrics for measuring KRaft performance
> > * KIP-902: Upgrade Zookeeper to 3.8.1
> > * KIP-917: Additional custom metadata for remote log segment
> >
> > Release notes for the 3.6.0 release:
> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhome.apache.org%2F~satishd%2Fkafka-3.6.0-rc1%2FRELEASE_NOTES.html=05%7C01%7Cgrussell%40vmware.com%7C573bc3210fd64ebf23e308dbb9f93c12%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C638308252383324078%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=qx8EKVV0QTKwU5GnqrJNSppZYr2UnKeZ3H%2BPGmuj3QM%3D=0
> >
> > *** Please download, test and vote by Saturday, September 23, 8am PT
> >
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > 

Re: [kafka-clients] [VOTE] 3.6.0 RC1

[Gary Russell]

I have lowered the priority because I found a work around, but it's still bad 
practice for a public class to depend on a class that is not published anywhere.

This is similar to [2], which has been open for months.

[1]: https://issues.apache.org/jira/browse/KAFKA-15482
[2]: https://issues.apache.org/jira/browse/KAFKA-15092

From: Gary Russell 
Sent: Wednesday, September 20, 2023 3:36 PM
To: us...@kafka.apache.org ; dev 
Subject: Re: [kafka-clients] [VOTE] 3.6.0 RC1

This is a blocker for me [1].

https://issues.apache.org/jira/browse/KAFKA-15482


From: Greg Harris 
Sent: Wednesday, September 20, 2023 12:45 PM
To: us...@kafka.apache.org 
Cc: dev ; kafka-clie...@googlegroups.com 

Subject: Re: [kafka-clients] [VOTE] 3.6.0 RC1

!! External Email

Hi all,

I verified the functionality of KIP-898 and the recent fix for
KAFKA-15473 with the following steps:

1. I started a 3.5.1 broker, and a 3.5.1 worker with most (>400)
publicly available plugins installed
2. I captured the output of /connector-plugins
3. I upgraded the worker to 3.6.0-rc1
4. I captured the output of /connector-plugins with various settings
of plugin.discovery
5. I ran the migration script to add manifests to my plugins
6. I captured the output of /connector-plugins with various settings
of plugin.discovery
7. I downgraded the worker to 3.5.1
8. I diffed the output of /connector-plugins across the different
cases and observed the expected changes.
a. When plugins are migrated for 3.6.0, all modes produce identical results.
b. When plugins are not migrated for 3.6.0, only_scan and
hybrid_warn produce identical results, hybrid_fail crashes, and
service_load is missing plugins
c. When upgrading from 3.5.1 I see that plugins with invalid
constructors are hidden, AK plugins now have versions, multi-interface
plugins now show each interface type, and plugins using AppInfoParser
change versions.
d. The startup logs now include descriptive errors for invalid
plugins that otherwise would have been thrown at runtime
d. The fix for KAFKA-15473 prevents duplicates
e. The output for 3.5.1 after downgrading is identical to before.

+1 (non-binding)

Thanks Satish for running the release!

On Wed, Sep 20, 2023 at 8:36 AM Divij Vaidya  wrote:
>
> Hey Satish
>
> My comments about documentation misses from RC0 vote thread [1] are
> still not addressed (such as missing metric documentation, formatting
> problems etc). Could you please mention why we shouldn't consider them
> as blockers to make RC1 as the final release?
>
> [1] 
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Fcokoxzd0jtgjtrlxoq7kkzmvpm75381t=05%7C01%7Cgrussell%40vmware.com%7C573bc3210fd64ebf23e308dbb9f93c12%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C638308252383167831%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=RTJ3LZ8uWUsTsxwMMO5B6izLcG1r%2F9il%2F98kvBkjz2Y%3D=0
>
> On Wed, Sep 20, 2023 at 4:53 PM Satish Duggana  
> wrote:
> >
> > Hello Kafka users, developers and client-developers,
> >
> > This is the second candidate for the release of Apache Kafka 3.6.0. Some of 
> > the major features include:
> >
> > * KIP-405 : Kafka Tiered Storage
> > * KIP-868 : KRaft Metadata Transactions
> > * KIP-875: First-class offsets support in Kafka Connect
> > * KIP-898: Modernize Connect plugin discovery
> > * KIP-938: Add more metrics for measuring KRaft performance
> > * KIP-902: Upgrade Zookeeper to 3.8.1
> > * KIP-917: Additional custom metadata for remote log segment
> >
> > Release notes for the 3.6.0 release:
> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhome.apache.org%2F~satishd%2Fkafka-3.6.0-rc1%2FRELEASE_NOTES.html=05%7C01%7Cgrussell%40vmware.com%7C573bc3210fd64ebf23e308dbb9f93c12%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C638308252383324078%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=qx8EKVV0QTKwU5GnqrJNSppZYr2UnKeZ3H%2BPGmuj3QM%3D=0
> >
> > *** Please download, test and vote by Saturday, September 23, 8am PT
> >
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fkafka.apache.org%2FKEYS=05%7C01%7Cgrussell%40vmware.com%7C573bc3210fd64ebf23e308dbb9f93c12%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C638308252383324078%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=vs2aCLRV%2Fha5yLjvuExhj8CQ45rwM0M%2BZHSqOVQU2i4%3D=0
> >
> > * Release artifacts to be voted upon (source and binary):
> > 

Re: [kafka-clients] [VOTE] 3.6.0 RC1

[Gary Russell]

This is a blocker for me [1].

https://issues.apache.org/jira/browse/KAFKA-15482


From: Greg Harris 
Sent: Wednesday, September 20, 2023 12:45 PM
To: us...@kafka.apache.org 
Cc: dev ; kafka-clie...@googlegroups.com 

Subject: Re: [kafka-clients] [VOTE] 3.6.0 RC1

!! External Email

Hi all,

I verified the functionality of KIP-898 and the recent fix for
KAFKA-15473 with the following steps:

1. I started a 3.5.1 broker, and a 3.5.1 worker with most (>400)
publicly available plugins installed
2. I captured the output of /connector-plugins
3. I upgraded the worker to 3.6.0-rc1
4. I captured the output of /connector-plugins with various settings
of plugin.discovery
5. I ran the migration script to add manifests to my plugins
6. I captured the output of /connector-plugins with various settings
of plugin.discovery
7. I downgraded the worker to 3.5.1
8. I diffed the output of /connector-plugins across the different
cases and observed the expected changes.
a. When plugins are migrated for 3.6.0, all modes produce identical results.
b. When plugins are not migrated for 3.6.0, only_scan and
hybrid_warn produce identical results, hybrid_fail crashes, and
service_load is missing plugins
c. When upgrading from 3.5.1 I see that plugins with invalid
constructors are hidden, AK plugins now have versions, multi-interface
plugins now show each interface type, and plugins using AppInfoParser
change versions.
d. The startup logs now include descriptive errors for invalid
plugins that otherwise would have been thrown at runtime
d. The fix for KAFKA-15473 prevents duplicates
e. The output for 3.5.1 after downgrading is identical to before.

+1 (non-binding)

Thanks Satish for running the release!

On Wed, Sep 20, 2023 at 8:36 AM Divij Vaidya  wrote:
>
> Hey Satish
>
> My comments about documentation misses from RC0 vote thread [1] are
> still not addressed (such as missing metric documentation, formatting
> problems etc). Could you please mention why we shouldn't consider them
> as blockers to make RC1 as the final release?
>
> [1] 
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Fcokoxzd0jtgjtrlxoq7kkzmvpm75381t=05%7C01%7Cgrussell%40vmware.com%7C573bc3210fd64ebf23e308dbb9f93c12%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C638308252383167831%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=RTJ3LZ8uWUsTsxwMMO5B6izLcG1r%2F9il%2F98kvBkjz2Y%3D=0
>
> On Wed, Sep 20, 2023 at 4:53 PM Satish Duggana  
> wrote:
> >
> > Hello Kafka users, developers and client-developers,
> >
> > This is the second candidate for the release of Apache Kafka 3.6.0. Some of 
> > the major features include:
> >
> > * KIP-405 : Kafka Tiered Storage
> > * KIP-868 : KRaft Metadata Transactions
> > * KIP-875: First-class offsets support in Kafka Connect
> > * KIP-898: Modernize Connect plugin discovery
> > * KIP-938: Add more metrics for measuring KRaft performance
> > * KIP-902: Upgrade Zookeeper to 3.8.1
> > * KIP-917: Additional custom metadata for remote log segment
> >
> > Release notes for the 3.6.0 release:
> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhome.apache.org%2F~satishd%2Fkafka-3.6.0-rc1%2FRELEASE_NOTES.html=05%7C01%7Cgrussell%40vmware.com%7C573bc3210fd64ebf23e308dbb9f93c12%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C638308252383324078%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=qx8EKVV0QTKwU5GnqrJNSppZYr2UnKeZ3H%2BPGmuj3QM%3D=0
> >
> > *** Please download, test and vote by Saturday, September 23, 8am PT
> >
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fkafka.apache.org%2FKEYS=05%7C01%7Cgrussell%40vmware.com%7C573bc3210fd64ebf23e308dbb9f93c12%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C638308252383324078%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=vs2aCLRV%2Fha5yLjvuExhj8CQ45rwM0M%2BZHSqOVQU2i4%3D=0
> >
> > * Release artifacts to be voted upon (source and binary):
> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhome.apache.org%2F~satishd%2Fkafka-3.6.0-rc1%2F=05%7C01%7Cgrussell%40vmware.com%7C573bc3210fd64ebf23e308dbb9f93c12%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C638308252383324078%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=mdPchqNg%2FvMeMCNetmJA5TVdafqHPfk5ZEfxu%2Fhiclc%3D=0
> >
> > * Maven artifacts to be voted upon:
> > 

[jira] [Created] (KAFKA-15482) kafka.utils.TestUtils Depends on MockTime Which is Not in Any Jar

[Gary Russell (Jira)]

Gary Russell created KAFKA-15482:


 Summary: kafka.utils.TestUtils Depends on MockTime Which is Not in 
Any Jar
 Key: KAFKA-15482
 URL: https://issues.apache.org/jira/browse/KAFKA-15482
 Project: Kafka
  Issue Type: Bug
Affects Versions: 3.6.0
Reporter: Gary Russell


Commit 

7eea2a3908fdcee1627c18827e6dcb5ed0089fdd 

Moved it to server-commons, but it is not included in the jar.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-714: Client metrics and observability

[Jun Rao]

Hi, Andrew,

Thanks for updating the KIP and sorry for the late response. A few more
comments.

110. Another potential motivation is the multiple clients support. Some of
the places may not have good monitoring support for non-java clients.

111. OpenTelemetry Naming: We replace '-' with '.' for metric name and
replace '-' with '_' for attributes. Why is the inconsistency?

112. OTLP specification: Page is not found from the link.

113. "Defining standard and required metrics makes the monitoring and
troubleshooting of clients from various client types ": Incomplete sentence.

114. standard/required metrics
114.1 Do other clients need to implement those metrics with the exact same
names?
114.2 What happens if some of those metrics are missing from a client?

115. "KafkaConsumer: both the "heart beat" and application threads": We
have an ongoing effort to refactor the consumer threading model (
https://cwiki.apache.org/confluence/display/KAFKA/Consumer+threading+refactor+design).
Once this is done, PRC requests will only be made from the background
thread. Should this KIP follow the new model only?

116. 'The metrics should contain the reason for the client termination by
including the client.terminating metric with the label “reason” ...'. Hmm,
are we introducing a new metric client.terminating? If so, that needs to be
explicitly listed.

117. "As the metrics plugin may need to add additional metrics on top of
this the generic metrics receiver in the broker will not add these labels
but rely on the plugins to do so," The sentence doesn't read well.

118. "it is possible for the client to send at most one accepted
out-of-profile per connection before the rate-limiter kicks in": If we do
this, do we still need the Terminating flag in PushTelemetryRequestV0?

119. "If there is no client metrics receiver plugin configured on the
broker, it will respond to GetTelemetrySubscriptionsRequest with
RequestedMetrics set to Null and a -1 SubscriptionId. The client should
send a new GetTelemetrySubscriptionsRequest after the PushIntervalMs has
expired. This allows the metrics receiver to be enabled or disabled without
having to restart the broker or reset the client connection."
"no client metrics receiver plugin configured" is defined by no metric
reporter implementing the ClientTelemetry interface, right? In that case,
it would be useful to avoid the clients sending
GetTelemetrySubscriptionsRequest periodically to preserve the current
behavior.

120. GetTelemetrySubscriptionsResponseV0 and PushTelemetryRequestV0: Could
we list error codes for each?

121. "ClientTelemetryReceiver.ClientTelemetryReceiver This method may be
called from the request handling thread": Where else can this method be
called?

122. DescribeConfigs/AlterConfigs already exist. Are we changing the ACL?

Thanks,

Jun

On Mon, Jul 31, 2023 at 4:33 AM Andrew Schofield <
andrew_schofield_j...@outlook.com> wrote:

> Hi Milind,
> Thanks for your question.
>
> On reflection, I agree that INVALID_RECORD is most likely to be caused by a
> problem in the serialization in the client. I have changed the client
> action in this case
> to “Log an error and stop pushing metrics”.
>
> I have updated the KIP text accordingly.
>
> Thanks,
> Andrew
>
> > On 31 Jul 2023, at 12:09, Milind Luthra 
> wrote:
> >
> > Hi Andrew,
> > Thanks for the clarifications.
> >
> > About 2b:
> > In case a client has a bug while serializing, it might be difficult for
> the
> > client to recover from that without code changes. In that, it might be
> good
> > to just log the INVALID_RECORD as an error, and treat the error as fatal
> > for the client (only fatal in terms of sending the metrics, the client
> can
> > keep functioning otherwise). What do you think?
> >
> > Thanks
> > Milind
> >
> > On Mon, Jul 24, 2023 at 8:18 PM Andrew Schofield <
> > andrew_schofield_j...@outlook.com> wrote:
> >
> >> Hi Milind,
> >> Thanks for your questions about the KIP.
> >>
> >> 1) I did some archaeology and looked at historical versions of the KIP.
> I
> >> think this is
> >> just a mistake. 5 minutes is the default metric push interval. 30
> minutes
> >> is a mystery
> >> to me. I’ve updated the KIP.
> >>
> >> 2) I think there are two situations in which INVALID_RECORD might occur.
> >> a) The client might perhaps be using a content-type that the broker does
> >> not support.
> >> The KIP mentions content-type as a future extension, but there’s only
> one
> >> supported
> >> to start with. Until we have multiple content-types, this seems out of
> >> scope. I think a
> >> future KIP would add another error code for this.
> >> b) The client might perhaps have a bug which means the metrics payload
> is
> >> malformed.
> >> Logging a warning and attempting the next metrics push on the push
> >> interval seems
> >> appropriate.
> >>
> >> UNKNOWN_SUBSCRIPTION_ID would indeed be handled by making an immediate
> >> GetTelemetrySubscriptionsRequest.
> >>
> >> UNSUPPORTED_COMPRESSION_TYPE seems like either 

Re: [DISCUSS] KIP-974 Docker Image for GraalVM based Native Kafka Broker

[Ismael Juma]

Hi Krishna,

Thanks for the KIP. The results are impressive! A few quick comments:

1. I noticed that the graalvm-reachability-metadata project now includes
kafka-clients[1]. Worth checking if the metadata over there matches what
we've observed for our own attempt.
2. Can we please add peak memory usage as one of the columns of the table?
I see that the information is covered elsewhere - it would be nice to have
it as part of the table too.
3. GraalVM for JDK 21 was recently released, can we use that? Does it
improve the numbers in a meaningful way?
4. Can we talk a bit more about the compatibility guarantees while this
image is still experimental?
5. It would be good to include details on the naming - how will this image
be named to differentiate itself from the non graalvm one?

[1]
https://github.com/oracle/graalvm-reachability-metadata/tree/df53148e4b2f730ad692daa956d72ab0cd7d43e3/metadata/org.apache.kafka/kafka-clients/3.5.1

Ismael

On Fri, Sep 8, 2023 at 12:17 AM Krishna Agarwal <
krishna0608agar...@gmail.com> wrote:

> Hi,
> I want to submit a KIP to deliver an experimental Apache Kafka docker
> image.
> The proposed docker image can launch brokers with sub-second startup time
> and minimal memory footprint by leveraging a GraalVM based native Kafka
> binary.
>
> KIP-974: Docker Image for GraalVM based Native Kafka Broker
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-974%3A+Docker+Image+for+GraalVM+based+Native+Kafka+Broker
> >
>
> Regards,
> Krishna
>

Re: [DISCUSS] KIP-974 Docker Image for GraalVM based Native Kafka Broker

[Ismael Juma]

Hi Federico,

I suggest taking this conversation to KIP-975 and keeping this one to the
graalvm specifics.

Ismael

On Fri, Sep 8, 2023 at 2:38 AM Federico Valeri  wrote:

> Hi Krishna, thanks for opening this discussion.
>
> I see you created two separate KIPs (974 and 975), but there are some
> common points (build system and test plan).
>
> Currently, the Docker image used for system tests is only supported in
> that limited scope, so the maintenance burden is minimal. Providing
> official Kafka images would be much more complicated. Have you
> considered how the image rebuild process would work in case a high
> severity CVE comes out for a non Kafka image dependency? In that case,
> there will be no Kafka release.
>
> Br
> Fede
>
> On Fri, Sep 8, 2023 at 9:17 AM Krishna Agarwal
>  wrote:
> >
> > Hi,
> > I want to submit a KIP to deliver an experimental Apache Kafka docker
> image.
> > The proposed docker image can launch brokers with sub-second startup time
> > and minimal memory footprint by leveraging a GraalVM based native Kafka
> > binary.
> >
> > KIP-974: Docker Image for GraalVM based Native Kafka Broker
> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-974%3A+Docker+Image+for+GraalVM+based+Native+Kafka+Broker
> >
> >
> > Regards,
> > Krishna
>

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

[Ismael Juma]

Hi Krishna,

Thanks for the KIP. A few quick questions:

1. Since this will only be available for Kafka 3.7 in the best case, I
suggest we go with Java 21 instead of Java 17. Also, we should be clear
about Java version expectations. Are we allowed to change the Java version
whenever we want? Or can it only be done in major versions? I prefer the
former option by default, but a subset of users may have their own plugins
running within the broker. Such users may need to have control over the
Java version they use - how would they achieve tha
2. We state that ubi-minimal would be the base image - it would be good to
provide some details on why this was the choice.
3. How does this docker image compare to the existing popular images?
Perhaps we can compare with the 3 most popular images for Apache Kafka
these days.

Ismael

On Fri, Sep 8, 2023 at 12:57 AM Krishna Agarwal <
krishna0608agar...@gmail.com> wrote:

> Hi,
> Apache Kafka does not have an official docker image currently.
> I want to submit a KIP to publish a docker image for Apache Kafka.
>
> KIP-975: Docker Image for Apache Kafka
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka
> >
>
> Regards,
> Krishna
>

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

[Ismael Juma]

Hi Viktor,

Thanks for the clarification. This is a separate KIP because the scope is
simpler than the graalvm docker images - there is much more to figure out
when it comes to using Kafka with graalvm.

Ismael

On Tue, Sep 19, 2023 at 3:12 AM Viktor Somogyi-Vass
 wrote:

> Hi Ismael,
>
> I'm not trying to advocate against the docker image, I just pointed out
> that the current scoping of the KIP may be a bit too generic and thought
> that KIP-974 and KIP-975 were aiming for mostly the same thing and can be
> discussed under one umbrella. Apologies if this was rooted in a
> misunderstanding.
>
> Kirshna,
>
> I think we need to refine the KIP a bit more. I think there are some
> interfaces that we need to include in the KIP as Kafka has plugins in
> certain cases where users are expected to provide implementation and I
> think it's worth discussing this in the KIP as they're kind of interfaces
> for users. Here are my questions in order:
> 1. In what environments do you want the image to be used? As I understand
> it would replace the current testing image and serve as a basis for
> development, but would it aim at production use cases too (docker-compose,
> Kubernetes, etc.)?
> 2. How do you plan to forward configs to the broker? Do we expect a
> populated server.properties file placed in a certain location or should the
> docker image create this file based on some input (like env vars)?
> 3. Certain parts can be pluggable, like metric reporters or remote log
> implementations that were just introduced by KIP-405. These manifest in jar
> files that must be put on the classpath of Kafka while certain classnames
> have to be configured. How do you plan to implement this, how do we
> allow users to configure such things?
>
> Thanks,
> Viktor
>
>
>
>
> On Thu, Sep 14, 2023 at 4:59 PM Kenneth Eversole
>  wrote:
>
> > Hello,
> >
> > I think this would be a wonderful improvement to the ecosystem. While
> > Viktor is correct that most Docker pipelines eventually lead to a
> > kubernetes deployment, that should not stop us from creating an
> > Official Docker Image. Creating a Docker image would allow us to ensure a
> > level of quality and support for people who want to deploy Kafka as a
> > container on baremetal machines, it could allow us to create
> > a sandbox/developer environment for new contributors and developers to
> test
> > and have a single agreed upon environment that kafka works in for future
> > KIPs and would most likely spawn more contributions from people wanting
> to
> > optimize kafka for k8s.
> >
> >
> > I am 100% for this and will gladly help if approved.
> >
> > Kenneth
> >
> > On Thu, Sep 14, 2023 at 5:47 AM Ismael Juma  wrote:
> >
> > > Hi Viktor,
> > >
> > > I disagree. Docker is a very popular deployment tool and it's not only
> > used
> > > with Kubernetes.
> > >
> > > Ismael
> > >
> > > On Thu, Sep 14, 2023, 1:14 AM Viktor Somogyi-Vass
> > >  wrote:
> > >
> > > > Hi Krishna,
> > > >
> > > > I think you should merge this KIP and KIP-974
> >  as there are
> > overlaps as
> > > > Federico pointed out on KIP-974
> > . I think you
> > should keep that one as it
> > > > has well defined goals (improve tests) while I feel this one is too
> > > > generic. Docker is usually just a tool for either testing or
> > Kubernetes,
> > > so
> > > > they have very well defined use-cases. In the case of Flink for
> > instance
> > > > the image is used for its kubernetes operator. The use case would
> > > determine
> > > > a lot of things and I think a generic image would likely not fit the
> > > needs
> > > > of all use-cases.
> > > >
> > > > Best,
> > > > Viktor
> > > >
> > > > On Fri, Sep 8, 2023 at 9:58 AM Krishna Agarwal <
> > > > krishna0608agar...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi,
> > > > > Apache Kafka does not have an official docker image currently.
> > > > > I want to submit a KIP to publish a docker image for Apache Kafka.
> > > > >
> > > > > KIP-975  >:
> > Docker Image for Apache Kafka
> > > > > <
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka
> > > > > >
> > > > >
> > > > > Regards,
> > > > > Krishna
> > > > >
> > > >
> > >
> >
>

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.6 #57

[Apache Jenkins Server]

See 

[DISCUSS] KIP-982: Access SslPrincipalMapper and kerberosShortNamer in Custom KafkaPrincipalBuilder

[Raghu B]

Hi everyone,

I would like to start the discussion on the KIP-982 to Access
SslPrincipalMapper and kerberosShortNamer in Custom KafkaPrincipalBuilder
https://cwiki.apache.org/confluence/display/KAFKA/KIP-982%3A+Access+SslPrincipalMapper+and+kerberosShortNamer+in+Custom+KafkaPrincipalBuilder

Looking forward to your feedback!

Thanks,
Raghu

Re: [kafka-clients] [VOTE] 3.6.0 RC1

[Greg Harris]

Hi all,

I verified the functionality of KIP-898 and the recent fix for
KAFKA-15473 with the following steps:

1. I started a 3.5.1 broker, and a 3.5.1 worker with most (>400)
publicly available plugins installed
2. I captured the output of /connector-plugins
3. I upgraded the worker to 3.6.0-rc1
4. I captured the output of /connector-plugins with various settings
of plugin.discovery
5. I ran the migration script to add manifests to my plugins
6. I captured the output of /connector-plugins with various settings
of plugin.discovery
7. I downgraded the worker to 3.5.1
8. I diffed the output of /connector-plugins across the different
cases and observed the expected changes.
a. When plugins are migrated for 3.6.0, all modes produce identical results.
b. When plugins are not migrated for 3.6.0, only_scan and
hybrid_warn produce identical results, hybrid_fail crashes, and
service_load is missing plugins
c. When upgrading from 3.5.1 I see that plugins with invalid
constructors are hidden, AK plugins now have versions, multi-interface
plugins now show each interface type, and plugins using AppInfoParser
change versions.
d. The startup logs now include descriptive errors for invalid
plugins that otherwise would have been thrown at runtime
d. The fix for KAFKA-15473 prevents duplicates
e. The output for 3.5.1 after downgrading is identical to before.

+1 (non-binding)

Thanks Satish for running the release!

On Wed, Sep 20, 2023 at 8:36 AM Divij Vaidya  wrote:
>
> Hey Satish
>
> My comments about documentation misses from RC0 vote thread [1] are
> still not addressed (such as missing metric documentation, formatting
> problems etc). Could you please mention why we shouldn't consider them
> as blockers to make RC1 as the final release?
>
> [1] https://lists.apache.org/thread/cokoxzd0jtgjtrlxoq7kkzmvpm75381t
>
> On Wed, Sep 20, 2023 at 4:53 PM Satish Duggana  
> wrote:
> >
> > Hello Kafka users, developers and client-developers,
> >
> > This is the second candidate for the release of Apache Kafka 3.6.0. Some of 
> > the major features include:
> >
> > * KIP-405 : Kafka Tiered Storage
> > * KIP-868 : KRaft Metadata Transactions
> > * KIP-875: First-class offsets support in Kafka Connect
> > * KIP-898: Modernize Connect plugin discovery
> > * KIP-938: Add more metrics for measuring KRaft performance
> > * KIP-902: Upgrade Zookeeper to 3.8.1
> > * KIP-917: Additional custom metadata for remote log segment
> >
> > Release notes for the 3.6.0 release:
> > https://home.apache.org/~satishd/kafka-3.6.0-rc1/RELEASE_NOTES.html
> >
> > *** Please download, test and vote by Saturday, September 23, 8am PT
> >
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > https://kafka.apache.org/KEYS
> >
> > * Release artifacts to be voted upon (source and binary):
> > https://home.apache.org/~satishd/kafka-3.6.0-rc1/
> >
> > * Maven artifacts to be voted upon:
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >
> > * Javadoc:
> > https://home.apache.org/~satishd/kafka-3.6.0-rc1/javadoc/
> >
> > * Tag to be voted upon (off 3.6 branch) is the 3.6.0 tag:
> > https://github.com/apache/kafka/releases/tag/3.6.0-rc1
> >
> > * Documentation:
> > https://kafka.apache.org/36/documentation.html
> >
> > * Protocol:
> > https://kafka.apache.org/36/protocol.html
> >
> > * Successful Jenkins builds for the 3.6 branch:
> > There are a few runs of unit/integration tests. You can see the latest at 
> > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.6/. We will continue 
> > running a few more iterations.
> > System tests:
> > We will send an update once we have the results.
> >
> > Thanks,
> > Satish.
> >
> > --
> > You received this message because you are subscribed to the Google Groups 
> > "kafka-clients" group.
> > To unsubscribe from this group and stop receiving emails from it, send an 
> > email to kafka-clients+unsubscr...@googlegroups.com.
> > To view this discussion on the web visit 
> > https://groups.google.com/d/msgid/kafka-clients/CAM-aUZ%3DuJ-SKeVFtBZwBjhLHKw4CbxF_ws%2BvQqaymGHFsC%2Bmdg%40mail.gmail.com.

[GitHub] [kafka-site] satishd commented on a diff in pull request #547: Added a blog entry for 3.6.0 release

[via GitHub]

satishd commented on code in PR #547:
URL: https://github.com/apache/kafka-site/pull/547#discussion_r1331784605


##
blog.html:
##
@@ -22,6 +22,46 @@
 
 
 Blog
+
+
+
+Apache 
Kafka 3.6.0 Release Announcement
+
+15 Sep 2023 - Satish Duggana (https://twitter.com/0xeed;>@SatishDuggana)
+We are proud to announce the release of Apache Kafka 3.6.0. 
This release contains many new features and improvements. This blog post will 
highlight some of the more prominent features. For a full list of changes, be 
sure to check the https://downloads.apache.org/kafka/3.6.0/RELEASE_NOTES.html;>release 
notes.
+See the https://kafka.apache.org/36/documentation.html#upgrade_3_6_0;>Upgrading 
to 3.6.0 from any version 0.8.x through 3.5.x section in the documentation 
for the list of notable changes and detailed upgrade steps.
+The ability to migrate Kafka clusters from ZK to KRaft mode 
with no downtime is still an early access feature. It is currently only 
suitable for testing in non production environments. See https://cwiki.apache.org/confluence/display/KAFKA/KIP-866+ZooKeeper+to+KRaft+Migration;>KIP-866
 for more details.

Review Comment:
   I meant delegation token support as part of KRaft related updates, including 
the migration. Added the blurb as suggested. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] satishd commented on pull request #547: Added a blog entry for 3.6.0 release

[via GitHub]

satishd commented on PR #547:
URL: https://github.com/apache/kafka-site/pull/547#issuecomment-1727830289

   @mumrah @divijvaidya  Thanks for the review. Addressed the review comments 
with the latest commits. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] divijvaidya commented on a diff in pull request #547: Added a blog entry for 3.6.0 release

[via GitHub]

divijvaidya commented on code in PR #547:
URL: https://github.com/apache/kafka-site/pull/547#discussion_r1331252329


##
blog.html:
##
@@ -22,6 +22,46 @@
 
 
 Blog
+
+
+
+Apache 
Kafka 3.6.0 Release Announcement
+
+15 Sep 2023 - Satish Duggana (https://twitter.com/0xeed;>@SatishDuggana)
+We are proud to announce the release of Apache Kafka 3.6.0. 
This release contains many new features and improvements. This blog post will 
highlight some of the more prominent features. For a full list of changes, be 
sure to check the https://downloads.apache.org/kafka/3.6.0/RELEASE_NOTES.html;>release 
notes.
+See the https://kafka.apache.org/36/documentation.html#upgrade_3_6_0;>Upgrading 
to 3.6.0 from any version 0.8.x through 3.5.x section in the documentation 
for the list of notable changes and detailed upgrade steps.
+The ability to migrate Kafka clusters from ZK to KRaft mode 
with no downtime is still an early access feature. It is currently only 
suitable for testing in non production environments. See https://cwiki.apache.org/confluence/display/KAFKA/KIP-866+ZooKeeper+to+KRaft+Migration;>KIP-866
 for more details.

Review Comment:
   Correct me I am wrong but delegation token support is different from Zk to 
KRaft migration. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [kafka-clients] [VOTE] 3.6.0 RC1

[Divij Vaidya]

Hey Satish

My comments about documentation misses from RC0 vote thread [1] are
still not addressed (such as missing metric documentation, formatting
problems etc). Could you please mention why we shouldn't consider them
as blockers to make RC1 as the final release?

[1] https://lists.apache.org/thread/cokoxzd0jtgjtrlxoq7kkzmvpm75381t

On Wed, Sep 20, 2023 at 4:53 PM Satish Duggana  wrote:
>
> Hello Kafka users, developers and client-developers,
>
> This is the second candidate for the release of Apache Kafka 3.6.0. Some of 
> the major features include:
>
> * KIP-405 : Kafka Tiered Storage
> * KIP-868 : KRaft Metadata Transactions
> * KIP-875: First-class offsets support in Kafka Connect
> * KIP-898: Modernize Connect plugin discovery
> * KIP-938: Add more metrics for measuring KRaft performance
> * KIP-902: Upgrade Zookeeper to 3.8.1
> * KIP-917: Additional custom metadata for remote log segment
>
> Release notes for the 3.6.0 release:
> https://home.apache.org/~satishd/kafka-3.6.0-rc1/RELEASE_NOTES.html
>
> *** Please download, test and vote by Saturday, September 23, 8am PT
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> https://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> https://home.apache.org/~satishd/kafka-3.6.0-rc1/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>
> * Javadoc:
> https://home.apache.org/~satishd/kafka-3.6.0-rc1/javadoc/
>
> * Tag to be voted upon (off 3.6 branch) is the 3.6.0 tag:
> https://github.com/apache/kafka/releases/tag/3.6.0-rc1
>
> * Documentation:
> https://kafka.apache.org/36/documentation.html
>
> * Protocol:
> https://kafka.apache.org/36/protocol.html
>
> * Successful Jenkins builds for the 3.6 branch:
> There are a few runs of unit/integration tests. You can see the latest at 
> https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.6/. We will continue 
> running a few more iterations.
> System tests:
> We will send an update once we have the results.
>
> Thanks,
> Satish.
>
> --
> You received this message because you are subscribed to the Google Groups 
> "kafka-clients" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to kafka-clients+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/kafka-clients/CAM-aUZ%3DuJ-SKeVFtBZwBjhLHKw4CbxF_ws%2BvQqaymGHFsC%2Bmdg%40mail.gmail.com.

[GitHub] [kafka-site] satishd commented on a diff in pull request #547: Added a blog entry for 3.6.0 release

[via GitHub]

satishd commented on code in PR #547:
URL: https://github.com/apache/kafka-site/pull/547#discussion_r1331713531


##
blog.html:
##
@@ -22,6 +22,46 @@
 
 
 Blog
+
+
+
+Apache 
Kafka 3.6.0 Release Announcement
+
+15 Sep 2023 - Satish Duggana (https://twitter.com/0xeed;>@SatishDuggana)
+We are proud to announce the release of Apache Kafka 3.6.0. 
This release contains many new features and improvements. This blog post will 
highlight some of the more prominent features. For a full list of changes, be 
sure to check the https://downloads.apache.org/kafka/3.6.0/RELEASE_NOTES.html;>release 
notes.
+See the https://kafka.apache.org/36/documentation.html#upgrade_3_6_0;>Upgrading 
to 3.6.0 from any version 0.8.x through 3.5.x section in the documentation 
for the list of notable changes and detailed upgrade steps.
+The ability to migrate Kafka clusters from ZK to KRaft mode 
with no downtime is still an early access feature. It is currently only 
suitable for testing in non production environments. See https://cwiki.apache.org/confluence/display/KAFKA/KIP-866+ZooKeeper+to+KRaft+Migration;>KIP-866
 for more details.
+https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage;>Tiered
 Storage is an early access feature. It is currently only suitable for 
testing in non production environments. See https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Tiered+Storage+Early+Access+Release+Notes;>Early
 Access Notes for more details.
+
+Note: ZooKeeper is marked as deprecated since 3.5.0 
release. ZooKeeper is planned to be removed in Apache Kafka 4.0. (Cf ZooKeeper Deprecation)
+Kafka Broker, Controller, Producer, Consumer and Admin 
Client
+
+KIP-405: Kafka Tiered Storagehttps://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage;>KIP-405.
 It introduces tiered storage feature in Kafka that provides separation of 
computation and storage in the broker.
+KIP-797: Accept duplicate listener on port for 
IPv4/IPv6https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=195726330;>KIP-797.
 Brokers can be configured with listeners that have same port on different ip 
stack like ipv4 and ipv6.
+KIP-863: Reduce CompletedFetch#parseRecord() memory 
copyhttps://cwiki.apache.org/confluence/pages/viewpage.action?pageId=225152035;>KIP-863.
 Reduced CompletedFetch#parseRecord() memory copy by deserializing using byte 
buffers.
+KIP-868: Metadata Transactionshttps://cwiki.apache.org/confluence/display/KAFKA/KIP-868+Metadata+Transactions;>KIP-868.
 This feature is about allowing the controller to generate atomic transactions 
of records that can exceed the maximum batch size.
+KIP-902: Upgrade Zookeeper to 3.8.2https://cwiki.apache.org/confluence/display/KAFKA/KIP-902%3A+Upgrade+Zookeeper+to+3.8.1;>KIP-902.
 Zookeeper client upgraded to 3.8.2 version as the current zookeeper dependency 
version 3.6.3 reached the end of life.
+KIP-917: Additional custom metadata for remote log 
segmenthttps://cwiki.apache.org/confluence/display/KAFKA/KIP-917%3A+Additional+custom+metadata+for+remote+log+segment;>KIP-917.
 Custom metedata support for remote log segments.
+KIP-930: Rename ambiguous Tiered Storage Metricshttps://cwiki.apache.org/confluence/display/KAFKA/KIP-930%3A+Rename+ambiguous+Tiered+Storage+Metrics;>KIP-930.
 Renamed ambiguous tiered storage metrics.
+KIP-937: Improve Message Timestamp Validationhttps://cwiki.apache.org/confluence/display/KAFKA/KIP-937%3A+Improve+Message+Timestamp+Validation;>KIP-937.
 Improved Producer's record timestamp validation.
+KIP-938: Add more metrics for measuring KRaft 
performancehttps://cwiki.apache.org/confluence/display/KAFKA/KIP-938%3A+Add+more+metrics+for+measuring+KRaft+performance;>KIP-938.
 Added the targeted KRaft performance metrics mentioned in KIP-938 except 
ForwardingManager metrics.
+
+Kafka Streams
+
+KIP-923: Add A Grace Period to Stream Table Join: 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-923%3A+Add+A+Grace+Period+to+Stream+Table+Join;>KIP-923Added
 a grace period to Stream Table join.
+KIP-941: Range queries to accept null lower and upper 
boundshttps://cwiki.apache.org/confluence/display/KAFKA/KIP-941%3A+Range+queries+to+accept+null+lower+and+upper+bounds;>KIP-941
 Range queries accept lower and upper bounds as null values.
+
+Kafka Connect
+
+KIP-793: Allow sink connectors to be used with 
topic-mutating 

[VOTE] 3.6.0 RC1

[Satish Duggana]

Hello Kafka users, developers and client-developers,

This is the second candidate for the release of Apache Kafka 3.6.0. Some of
the major features include:

* KIP-405 : Kafka Tiered Storage
* KIP-868 : KRaft Metadata Transactions
* KIP-875: First-class offsets support in Kafka Connect
* KIP-898: Modernize Connect plugin discovery
* KIP-938: Add more metrics for measuring KRaft performance
* KIP-902: Upgrade Zookeeper to 3.8.1
* KIP-917: Additional custom metadata for remote log segment

Release notes for the 3.6.0 release:
https://home.apache.org/~satishd/kafka-3.6.0-rc1/RELEASE_NOTES.html

*** Please download, test and vote by Saturday, September 23, 8am PT

Kafka's KEYS file containing PGP keys we use to sign the release:
https://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
https://home.apache.org/~satishd/kafka-3.6.0-rc1/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/org/apache/kafka/

* Javadoc:
https://home.apache.org/~satishd/kafka-3.6.0-rc1/javadoc/

* Tag to be voted upon (off 3.6 branch) is the 3.6.0 tag:
https://github.com/apache/kafka/releases/tag/3.6.0-rc1

* Documentation:
https://kafka.apache.org/36/documentation.html

* Protocol:
https://kafka.apache.org/36/protocol.html

* Successful Jenkins builds for the 3.6 branch:
There are a few runs of unit/integration tests. You can see the latest at
https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.6/. We will continue
running a few more iterations.
System tests:
We will send an update once we have the results.

Thanks,
Satish.

[GitHub] [kafka-site] clolov commented on a diff in pull request #547: Added a blog entry for 3.6.0 release

[via GitHub]

clolov commented on code in PR #547:
URL: https://github.com/apache/kafka-site/pull/547#discussion_r1331664528


##
blog.html:
##
@@ -22,6 +22,46 @@
 
 
 Blog
+
+
+
+Apache 
Kafka 3.6.0 Release Announcement
+
+15 Sep 2023 - Satish Duggana (https://twitter.com/0xeed;>@SatishDuggana)
+We are proud to announce the release of Apache Kafka 3.6.0. 
This release contains many new features and improvements. This blog post will 
highlight some of the more prominent features. For a full list of changes, be 
sure to check the https://downloads.apache.org/kafka/3.6.0/RELEASE_NOTES.html;>release 
notes.
+See the https://kafka.apache.org/36/documentation.html#upgrade_3_6_0;>Upgrading 
to 3.6.0 from any version 0.8.x through 3.5.x section in the documentation 
for the list of notable changes and detailed upgrade steps.
+The ability to migrate Kafka clusters from ZK to KRaft mode 
with no downtime is still an early access feature. It is currently only 
suitable for testing in non production environments. See https://cwiki.apache.org/confluence/display/KAFKA/KIP-866+ZooKeeper+to+KRaft+Migration;>KIP-866
 for more details.
+https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage;>Tiered
 Storage is an early access feature. It is currently only suitable for 
testing in non production environments. See https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Tiered+Storage+Early+Access+Release+Notes;>Early
 Access Notes for more details.
+
+Note: ZooKeeper is marked as deprecated since 3.5.0 
release. ZooKeeper is planned to be removed in Apache Kafka 4.0. (Cf ZooKeeper Deprecation)
+Kafka Broker, Controller, Producer, Consumer and Admin 
Client
+
+KIP-405: Kafka Tiered Storagehttps://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage;>KIP-405.
 It introduces tiered storage feature in Kafka that provides separation of 
computation and storage in the broker.
+KIP-797: Accept duplicate listener on port for 
IPv4/IPv6https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=195726330;>KIP-797.
 Brokers can be configured with listeners that have same port on different ip 
stack like ipv4 and ipv6.
+KIP-863: Reduce CompletedFetch#parseRecord() memory 
copyhttps://cwiki.apache.org/confluence/pages/viewpage.action?pageId=225152035;>KIP-863.
 Reduced CompletedFetch#parseRecord() memory copy by deserializing using byte 
buffers.
+KIP-868: Metadata Transactionshttps://cwiki.apache.org/confluence/display/KAFKA/KIP-868+Metadata+Transactions;>KIP-868.
 This feature is about allowing the controller to generate atomic transactions 
of records that can exceed the maximum batch size.
+KIP-902: Upgrade Zookeeper to 3.8.1https://cwiki.apache.org/confluence/display/KAFKA/KIP-902%3A+Upgrade+Zookeeper+to+3.8.1;>KIP-902.
 Zookeeper client upgraded to 3.8.1 version as the current zookeeper dependency 
version 3.6.3 reached the end of life.

Review Comment:
   Updated



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: Re: [DISCUSS] KIP-951: Leader discovery optimisations for the client

[Crispin Bernier]

Hi,

I've updated the KIP with benchmark results focusing more directly on the
redirect case, please review and +1 in the voting thread if convinced.

Thank you,
Crispin

On Wed, Jul 26, 2023 at 11:13 PM Luke Chen  wrote:

> Thanks for adding the benchmark results, Crispin!
> IMO, 2~5% performance improvement is small, but given the change is small,
> cost is also small (only append endpoint info when NOT_LEADER_OR_FOLLOWER..
> etc error), I think it is worth doing it.
>
> Thank you.
> Luke
>
> On Wed, Jul 26, 2023 at 12:33 AM Ismael Juma  wrote:
>
> > Thanks Crispin!
> >
> > Ismael
> >
> > On Mon, Jul 24, 2023 at 8:16 PM Crispin Bernier
> >  wrote:
> >
> > > I updated the wiki to include both results along with their average.
> > >
> > > Thank you,
> > > Crispin
> > >
> > > On Mon, Jul 24, 2023 at 10:58 AM Ismael Juma 
> wrote:
> > >
> > > > Hi Crispin,
> > > >
> > > > One additional question, the wiki says "The results are averaged
> over 2
> > > > runs.". Can you please provide some measure of variance in the
> > > > distribution, i.e. were both results similar to each other for both
> > > cases?
> > > >
> > > > Ismael
> > > >
> > > > On Fri, Jul 21, 2023 at 11:31 AM Ismael Juma 
> > wrote:
> > > >
> > > > > Thanks for the update Crispin - very helpful to have actual
> > performance
> > > > > data. 2-5% for the default configuration is a bit on the low side
> for
> > > > this
> > > > > kind of proposal.
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Thu, Jul 20, 2023 at 11:33 PM Crispin Bernier
> > > > >  wrote:
> > > > >
> > > > >> Benchmark numbers have been posted on the KIP, please review.
> > > > >>
> > > > >> On 2023/07/20 13:03:00 Mayank Shekhar Narula wrote:
> > > > >> > Jun
> > > > >> >
> > > > >> > Thanks for the feedback.
> > > > >> >
> > > > >> > Numbers to follow.
> > > > >> >
> > > > >> > If we don't plan to
> > > > >> > > bump up the FetchResponse version, we could just remove the
> > > > reference
> > > > >> to
> > > > >> > > version 16.
> > > > >> >
> > > > >> > Fixed.
> > > > >> >
> > > > >> > On Thu, Jul 20, 2023 at 1:28 AM Jun Rao
> >  > > >
> > > > >> wrote:
> > > > >> >
> > > > >> > > Hi, Mayank,
> > > > >> > >
> > > > >> > > Thanks for the KIP. I agree with others that it would be
> useful
> > to
> > > > >> see the
> > > > >> > > performance results. Otherwise, just a minor comment. If we
> > don't
> > > > >> plan to
> > > > >> > > bump up the FetchResponse version, we could just remove the
> > > > reference
> > > > >> to
> > > > >> > > version 16.
> > > > >> > >
> > > > >> > > Jun
> > > > >> > >
> > > > >> > > On Wed, Jul 19, 2023 at 2:31 PM Mayank Shekhar Narula <
> > > > >> > > mayanks.nar...@gmail.com> wrote:
> > > > >> > >
> > > > >> > > > Luke
> > > > >> > > >
> > > > >> > > > Thanks for the interest in the KIP.
> > > > >> > > >
> > > > >> > > > But what if the consumer was fetching from the follower?
> > > > >> > > >
> > > > >> > > > We already include `PreferredReadReplica` in the fetch
> > response.
> > > > >> > > > > Should we put the node info of PreferredReadReplica under
> > this
> > > > >> case,
> > > > >> > > > > instead of the leader's info?
> > > > >> > > > >
> > > > >> > > >
> > > > >> > > > PreferredReadReplica is the decided on the leader. Looking
> at
> > > the
> > > > >> Java
> > > > >> > > > client code, AbstractFetch::selectReadReplica, first fetch
> > > request
> > > > >> goes
> > > > >> > > to
> > > > >> > > > Leader of the partition -> Sends back PreferredReadReplica
> ->
> > > Next
> > > > >> fetch
> > > > >> > > > uses PreferredReadReplica. So as long as leader is
> available,
> > > > >> > > > PreferredReadReplica would be found in subsequent fetches.
> > > > >> > > >
> > > > >> > > > Also, under this case, should we include the leader's info
> in
> > > the
> > > > >> > > response?
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > In this case, I think the follower would fail the fetch if
> it
> > > > knows
> > > > >> a
> > > > >> > > > different leader. If the follower knows a newer leader, it
> > would
> > > > >> return
> > > > >> > > new
> > > > >> > > > leader information in the response, for the client to act
> on.
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > Will we include the leader/node info in the response when
> > having
> > > > >> > > > > `UNKNOWN_LEADER_EPOCH` error?
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > My understanding is UNKNOWN_LEADER_EPOCH when a request
> from a
> > > > >> client
> > > > >> > > has a
> > > > >> > > > newer epoch than the broker. So the client is already up to
> > date
> > > > on
> > > > >> new
> > > > >> > > > leader information, it's the broker that has the catching up
> > to
> > > > do.
> > > > >> I
> > > > >> > > think
> > > > >> > > > there might be some optimisations to make sure the broker
> > > > refreshes
> > > > >> its
> > > > >> > > > metadata quickly, so it can quickly recover to handle
> requests
> > > > that
> > > > >> > > > previously returned UNKNOWN_LEADER_EPOCH. But this work 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.6 #55

[Apache Jenkins Server]

See 

[GitHub] [kafka-site] gaurav-narula commented on a diff in pull request #547: Added a blog entry for 3.6.0 release

[via GitHub]

gaurav-narula commented on code in PR #547:
URL: https://github.com/apache/kafka-site/pull/547#discussion_r1331193907


##
blog.html:
##
@@ -22,6 +22,46 @@
 
 
 Blog
+
+
+
+Apache 
Kafka 3.6.0 Release Announcement
+
+15 Sep 2023 - Satish Duggana (https://twitter.com/0xeed;>@SatishDuggana)
+We are proud to announce the release of Apache Kafka 3.6.0. 
This release contains many new features and improvements. This blog post will 
highlight some of the more prominent features. For a full list of changes, be 
sure to check the https://downloads.apache.org/kafka/3.6.0/RELEASE_NOTES.html;>release 
notes.
+See the https://kafka.apache.org/36/documentation.html#upgrade_3_6_0;>Upgrading 
to 3.6.0 from any version 0.8.x through 3.5.x section in the documentation 
for the list of notable changes and detailed upgrade steps.
+The ability to migrate Kafka clusters from ZK to KRaft mode 
with no downtime is still an early access feature. It is currently only 
suitable for testing in non production environments. See https://cwiki.apache.org/confluence/display/KAFKA/KIP-866+ZooKeeper+to+KRaft+Migration;>KIP-866
 for more details.
+https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage;>Tiered
 Storage is an early access feature. It is currently only suitable for 
testing in non production environments. See https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Tiered+Storage+Early+Access+Release+Notes;>Early
 Access Notes for more details.
+
+Note: ZooKeeper is marked as deprecated since 3.5.0 
release. ZooKeeper is planned to be removed in Apache Kafka 4.0. (Cf ZooKeeper Deprecation)
+Kafka Broker, Controller, Producer, Consumer and Admin 
Client
+
+KIP-405: Kafka Tiered Storagehttps://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage;>KIP-405.
 It introduces tiered storage feature in Kafka that provides separation of 
computation and storage in the broker.
+KIP-797: Accept duplicate listener on port for 
IPv4/IPv6https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=195726330;>KIP-797.
 Brokers can be configured with listeners that have same port on different ip 
stack like ipv4 and ipv6.
+KIP-863: Reduce CompletedFetch#parseRecord() memory 
copyhttps://cwiki.apache.org/confluence/pages/viewpage.action?pageId=225152035;>KIP-863.
 Reduced CompletedFetch#parseRecord() memory copy by deserializing using byte 
buffers.
+KIP-868: Metadata Transactionshttps://cwiki.apache.org/confluence/display/KAFKA/KIP-868+Metadata+Transactions;>KIP-868.
 This feature is about allowing the controller to generate atomic transactions 
of records that can exceed the maximum batch size.
+KIP-902: Upgrade Zookeeper to 3.8.2https://cwiki.apache.org/confluence/display/KAFKA/KIP-902%3A+Upgrade+Zookeeper+to+3.8.1;>KIP-902.
 Zookeeper client upgraded to 3.8.2 version as the current zookeeper dependency 
version 3.6.3 reached the end of life.
+KIP-917: Additional custom metadata for remote log 
segmenthttps://cwiki.apache.org/confluence/display/KAFKA/KIP-917%3A+Additional+custom+metadata+for+remote+log+segment;>KIP-917.
 Custom metedata support for remote log segments.
+KIP-930: Rename ambiguous Tiered Storage Metricshttps://cwiki.apache.org/confluence/display/KAFKA/KIP-930%3A+Rename+ambiguous+Tiered+Storage+Metrics;>KIP-930.
 Renamed ambiguous tiered storage metrics.
+KIP-937: Improve Message Timestamp Validationhttps://cwiki.apache.org/confluence/display/KAFKA/KIP-937%3A+Improve+Message+Timestamp+Validation;>KIP-937.
 Improved Producer's record timestamp validation.
+KIP-938: Add more metrics for measuring KRaft 
performancehttps://cwiki.apache.org/confluence/display/KAFKA/KIP-938%3A+Add+more+metrics+for+measuring+KRaft+performance;>KIP-938.
 Added the targeted KRaft performance metrics mentioned in KIP-938 except 
ForwardingManager metrics.
+
+Kafka Streams
+
+KIP-923: Add A Grace Period to Stream Table Join: 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-923%3A+Add+A+Grace+Period+to+Stream+Table+Join;>KIP-923Added
 a grace period to Stream Table join.
+KIP-941: Range queries to accept null lower and upper 
boundshttps://cwiki.apache.org/confluence/display/KAFKA/KIP-941%3A+Range+queries+to+accept+null+lower+and+upper+bounds;>KIP-941
 Range queries accept lower and upper bounds as null values.
+
+Kafka Connect
+
+KIP-793: Allow sink connectors to be used with 
topic-mutating 

Re: [VOTE] 3.6.0 RC0

[Satish Duggana]

Thanks Luke for reviewing and running the changes
https://github.com/apache/kafka/pull/14409 for merging into 3.6.

On Wed, 20 Sept 2023 at 06:09, Satish Duggana  wrote:
>
> Hi David,
> Thanks for the update. Please get it reviewed sooner. I plan to create
> RC1 in the next 5 hrs. As it is not a release blocker, I will merge it
> if this PR is approved and jenkins job looks good within that time.
>
> Thanks,
> Satish.
>
> On Wed, 20 Sept 2023 at 01:42, David Arthur
>  wrote:
> >
> > As part of validating this RC, I found that most of the ZK migration system
> > tests were failing. I investigated this today and found that it was just
> > problems in the test code. This is not a blocker, but since we're doing
> > RC1, I'd like to include it https://github.com/apache/kafka/pull/14409
> >
> > On Tue, Sep 19, 2023 at 12:57 PM Satish Duggana 
> > wrote:
> >
> > > Hi All,
> > > I will start creating a new RC with the fix and send it for a vote by
> > > tomorrow as https://issues.apache.org/jira/browse/KAFKA-15473 is
> > > determined to be a release blocker.
> > >
> > > Thanks.
> > > Satish.
> > >
> > > On Tue, 19 Sept 2023 at 22:26, Satish Duggana 
> > > wrote:
> > > >
> > > > Thanks Chris for your comment on the reported issue.
> > > >
> > > > ~Satish.
> > > >
> > > > On Tue, 19 Sept 2023 at 22:16, Chris Egerton 
> > > wrote:
> > > > >
> > > > > Hi all,
> > > > >
> > > > > Given the regression raised by Greg Harris (
> > > > > https://issues.apache.org/jira/browse/KAFKA-15473) on the release
> > > thread, I
> > > > > believe we should generate a new candidate with a fix.
> > > > >
> > > > > I'm -1 (binding) on this RC.
> > > > >
> > > > > Best,
> > > > >
> > > > > Chris
> > > > >
> > > > > On Tue, Sep 19, 2023 at 10:30 AM Christo Lolov  > > >
> > > > > wrote:
> > > > >
> > > > > > Heya,
> > > > > >
> > > > > > I have compiled and ran the test target successfully for the
> > > 3.6.0-rc0
> > > > > > branch on:
> > > > > >
> > > > > > Java 11, Scala 2.13 - ARM
> > > > > > Java 17, Scala 2.13 - ARM
> > > > > > Java 20, Scala 2.18 - ARM
> > > > > > Java 11, Scala 2.13 - Intel x86
> > > > > > Java 17, Scala 2.13 - Intel x86
> > > > > > Java 20, Scala 2.13 - Intel x86
> > > > > >
> > > > > > I will update the Zookeeper KIP title in the KIPs page, that's my
> > > miss
> > > > > >
> > > > > > Best,
> > > > > > Christo
> > > > > >
> > > > > > On Tue, 19 Sept 2023 at 13:21, Divij Vaidya  > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hey Satish
> > > > > > >
> > > > > > > Thank you for managing this release. I have a few comments:
> > > > > > >
> > > > > > > Documentation
> > > > > > >
> > > > > > > 1. Section: Zookeeper/Stable Version - The documentation states
> > > "The
> > > > > > > current stable branch is 3.5. Kafka is regularly updated to 
> > > > > > > include
> > > > > > > the latest release in the 3.5 series." in the ZooKeeper section.
> > > That
> > > > > > > needs an update since we are running Zk 3.8 now.
> > > > > > >
> > > > > > > 2. Section: Zookeeper/Migration - The documentation states
> > > "Migration
> > > > > > > of an existing ZooKeeper based Kafka cluster to KRaft is currently
> > > > > > > Preview and we expect it to be ready for production usage in
> > > version
> > > > > > > 3.6.". This probably needs an update on whether it is production
> > > ready
> > > > > > > or not in 3.6
> > > > > > >
> > > > > > > 3. Section: Kraft/missing features
> > > > > > > (https://kafka.apache.org/36/documentation.html#kraft_missing) - I
> > > > > > > believe that delegation token is now part of 3.6? I think this
> > > > > > > probably needs an update.
> > > > > > >
> > > > > > > 4. Section: Configuration/rack.aware.assignment.strategy - there
> > > seems
> > > > > > > to be a formatting problem starting from here
> > > > > > > (
> > > > > > >
> > > > > >
> > > https://kafka.apache.org/36/documentation.html#streamsconfigs_rack.aware.assignment.strategy
> > > > > > > )
> > > > > > >
> > > > > > > 5. Section: KRaft Monitoring - Newly added metrics in
> > > > > > > https://issues.apache.org/jira/browse/KAFKA-15183 are missing
> > > from the
> > > > > > > documentation here.
> > > > > > >
> > > > > > > Release notes
> > > > > > >
> > > > > > > 1. I found a bunch of tickets which have not been marked with a
> > > > > > > release version but have been resolved in last 6 months using the
> > > > > > > query
> > > > > > >
> > > > > >
> > > https://issues.apache.org/jira/browse/KAFKA-15380?jql=project%20%3D%20KAFKA%20AND%20status%20in%20(Resolved%2C%20Closed)%20AND%20resolution%20%3D%20Fixed%20AND%20fixVersion%20%3D%20EMPTY%20AND%20resolved%20%3E%3D%20-24w%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC
> > > > > > > . Are some of them targeted for 3.6 release?
> > > > > > >
> > > > > > > 2. The KIP "KIP-902: Upgrade Zookeeper to 3.8.1" should probably 
> > > > > > > be
> > > > > > > renamed to include 3.8.2 since code uses version 3.8.2 of
> > > Zookeeper.
> > > > > > >
> > > > > > >
> > > > > > > Additionally, I 

Re: [DISCUSS] KIP-980: Allow creating connectors in a stopped state

[Ashwin]

Thanks Yash! This is very useful for migrating connectors from one cluster
to another.

I had the following comments/questions

1. Is the offset read using `GET /offsets` api always guaranteed to be in a
format accepted by `PATCH /offsets` ?
2. I had to tackle a similar migration situation but the two connect
clusters in question were using the same backing Kafka cluster. The
challenge in this case is that when I delete the original connector, I want
to retain offsets and config topics. Do you think we should support
deletion of a connector without removal of these topics as part of this KIP
?
3. In the case of a downgrade, how will Connect worker handle the optional
“state” field in config topic ?

Thanks,
Ashwin




On Sun, Sep 17, 2023 at 11:09 PM Yash Mayya  wrote:

> Hi all,
>
> I'd like to begin discussion on a KIP to allow creating connectors in a
> stopped state -
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-980%3A+Allow+creating+connectors+in+a+stopped+state
>
>
> Thanks,
> Yash
>

[GitHub] [kafka-site] mumrah commented on a diff in pull request #547: Added a blog entry for 3.6.0 release

[via GitHub]

mumrah commented on code in PR #547:
URL: https://github.com/apache/kafka-site/pull/547#discussion_r1331519680


##
blog.html:
##
@@ -22,6 +22,46 @@
 
 
 Blog
+
+
+
+Apache 
Kafka 3.6.0 Release Announcement
+
+15 Sep 2023 - Satish Duggana (https://twitter.com/0xeed;>@SatishDuggana)
+We are proud to announce the release of Apache Kafka 3.6.0. 
This release contains many new features and improvements. This blog post will 
highlight some of the more prominent features. For a full list of changes, be 
sure to check the https://downloads.apache.org/kafka/3.6.0/RELEASE_NOTES.html;>release 
notes.
+See the https://kafka.apache.org/36/documentation.html#upgrade_3_6_0;>Upgrading 
to 3.6.0 from any version 0.8.x through 3.5.x section in the documentation 
for the list of notable changes and detailed upgrade steps.
+The ability to migrate Kafka clusters from ZK to KRaft mode 
with no downtime is still an early access feature. It is currently only 
suitable for testing in non production environments. See https://cwiki.apache.org/confluence/display/KAFKA/KIP-866+ZooKeeper+to+KRaft+Migration;>KIP-866
 for more details.
+https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage;>Tiered
 Storage is an early access feature. It is currently only suitable for 
testing in non production environments. See https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Tiered+Storage+Early+Access+Release+Notes;>Early
 Access Notes for more details.
+
+Note: ZooKeeper is marked as deprecated since 3.5.0 
release. ZooKeeper is planned to be removed in Apache Kafka 4.0. (Cf ZooKeeper Deprecation)

Review Comment:
   nit: `` tags should be ``



##
blog.html:
##
@@ -22,6 +22,46 @@
 
 
 Blog
+
+
+
+Apache 
Kafka 3.6.0 Release Announcement
+
+15 Sep 2023 - Satish Duggana (https://twitter.com/0xeed;>@SatishDuggana)
+We are proud to announce the release of Apache Kafka 3.6.0. 
This release contains many new features and improvements. This blog post will 
highlight some of the more prominent features. For a full list of changes, be 
sure to check the https://downloads.apache.org/kafka/3.6.0/RELEASE_NOTES.html;>release 
notes.
+See the https://kafka.apache.org/36/documentation.html#upgrade_3_6_0;>Upgrading 
to 3.6.0 from any version 0.8.x through 3.5.x section in the documentation 
for the list of notable changes and detailed upgrade steps.
+The ability to migrate Kafka clusters from ZK to KRaft mode 
with no downtime is still an early access feature. It is currently only 
suitable for testing in non production environments. See https://cwiki.apache.org/confluence/display/KAFKA/KIP-866+ZooKeeper+to+KRaft+Migration;>KIP-866
 for more details.

Review Comment:
   ```suggestion
   
   The ability to migrate Kafka clusters from a ZooKeeper 
metadata system to a KRaft metadata system is
   now considered stable and suitable for production 
environments. See the ZooKeeper to KRaft migration
   https://kafka.apache.org/documentation/#kraft_zk_migration;>operations 
documentation for
   details. Note that support for JBOD is still not 
available for KRaft clusters, therefor clusters
   utilizing JBOD cannot be migrated. See https://cwiki.apache.org/confluence/display/KAFKA/KIP-858%3A+Handle+JBOD+broker+disk+failure+in+KRaft;>KIP-858
   for details regarding KRaft and JBOD.
   
   ```
   
   Here's what I had in my other PR, let's just include it in this one.



##
blog.html:
##
@@ -22,6 +22,46 @@
 
 
 Blog
+
+
+
+Apache 
Kafka 3.6.0 Release Announcement
+
+15 Sep 2023 - Satish Duggana (https://twitter.com/0xeed;>@SatishDuggana)
+We are proud to announce the release of Apache Kafka 3.6.0. 
This release contains many new features and improvements. This blog post will 
highlight some of the more prominent features. For a full list of changes, be 
sure to check the https://downloads.apache.org/kafka/3.6.0/RELEASE_NOTES.html;>release 
notes.
+See the https://kafka.apache.org/36/documentation.html#upgrade_3_6_0;>Upgrading 
to 3.6.0 from any version 0.8.x through 3.5.x section in the documentation 
for the list of notable changes and detailed upgrade steps.
+The ability to migrate Kafka clusters from ZK to KRaft mode 
with no downtime is still an early access feature. It is currently only 

[jira] [Created] (KAFKA-15480) Add RemoteStorageInterruptedException

[Mital Awachat (Jira)]

Mital Awachat created KAFKA-15480:
-

 Summary: Add RemoteStorageInterruptedException
 Key: KAFKA-15480
 URL: https://issues.apache.org/jira/browse/KAFKA-15480
 Project: Kafka
  Issue Type: Task
  Components: core
Affects Versions: 3.6.0
Reporter: Mital Awachat


Introduce `RemoteStorageInterruptedException` to propagate interruptions from 
the plugin to Kafka without generated (false) errors. 

It allows the plugin to notify Kafka an API operation in progress was 
interrupted as a result of task cancellation, which can happen under changes 
such as leadership migration or topic deletion.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15481) Concurrency bug in RemoteIndexCache leads to IOException

[Divij Vaidya (Jira)]

Divij Vaidya created KAFKA-15481:


 Summary: Concurrency bug in RemoteIndexCache leads to IOException
 Key: KAFKA-15481
 URL: https://issues.apache.org/jira/browse/KAFKA-15481
 Project: Kafka
  Issue Type: Bug
Affects Versions: 3.6.0
Reporter: Divij Vaidya
 Fix For: 3.7.0


RemoteIndexCache has a concurrency bug which leads to IOException while 
fetching data from remote tier.

Below events in order of timeline -

Thread 1 (cache thread): invalidates the entry, removalListener is invoked 
async, so the files have not been renamed to "deleted" suffix yet.

Thread 2: (fetch thread): tries to find entry in cache, doesn't find it because 
it has been removed by 1, fetches the entry from S3, writes it to existing file 
(using replace existing)

Thread 1: async removalListener is invoked, acquires a lock on old entry (which 
has been removed from cache), it renames the file to "deleted" and starts 
deleting it

Thread 2: Tries to create in-memory/mmapped index, but doesn't find the file 
and hence, creates a new file of size 2GB in AbstractIndex constructor. JVM 
returns an error as it won't allow creation of 2GB random access file.

*Potential Fix*
Use EvictionListener instead of RemovalListener in Caffeine cache as per the 
documentation:
{quote} When the operation must be performed synchronously with eviction, use 
{{Caffeine.evictionListener(RemovalListener)}} instead. This listener will only 
be notified when {{RemovalCause.wasEvicted()}} is true. For an explicit 
removal, {{Cache.asMap()}} offers compute methods that are performed 
atomically.{quote}
This will ensure that removal from cache and marking the file with delete 
suffix is synchronously done, hence the above race condition will not occur.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[GitHub] [kafka-site] mumrah commented on a diff in pull request #547: Added a blog entry for 3.6.0 release

[via GitHub]

mumrah commented on code in PR #547:
URL: https://github.com/apache/kafka-site/pull/547#discussion_r1331517587


##
blog.html:
##
@@ -22,6 +22,46 @@
 
 
 Blog
+
+
+
+Apache 
Kafka 3.6.0 Release Announcement
+
+15 Sep 2023 - Satish Duggana (https://twitter.com/0xeed;>@SatishDuggana)
+We are proud to announce the release of Apache Kafka 3.6.0. 
This release contains many new features and improvements. This blog post will 
highlight some of the more prominent features. For a full list of changes, be 
sure to check the https://downloads.apache.org/kafka/3.6.0/RELEASE_NOTES.html;>release 
notes.
+See the https://kafka.apache.org/36/documentation.html#upgrade_3_6_0;>Upgrading 
to 3.6.0 from any version 0.8.x through 3.5.x section in the documentation 
for the list of notable changes and detailed upgrade steps.
+The ability to migrate Kafka clusters from ZK to KRaft mode 
with no downtime is still an early access feature. It is currently only 
suitable for testing in non production environments. See https://cwiki.apache.org/confluence/display/KAFKA/KIP-866+ZooKeeper+to+KRaft+Migration;>KIP-866
 for more details.
+https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage;>Tiered
 Storage is an early access feature. It is currently only suitable for 
testing in non production environments. See https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Tiered+Storage+Early+Access+Release+Notes;>Early
 Access Notes for more details.
+
+Note: ZooKeeper is marked as deprecated since 3.5.0 
release. ZooKeeper is planned to be removed in Apache Kafka 4.0. (Cf ZooKeeper Deprecation)
+Kafka Broker, Controller, Producer, Consumer and Admin 
Client
+
+KIP-405: Kafka Tiered Storagehttps://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage;>KIP-405.
 It introduces tiered storage feature in Kafka that provides separation of 
computation and storage in the broker.
+KIP-797: Accept duplicate listener on port for 
IPv4/IPv6https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=195726330;>KIP-797.
 Brokers can be configured with listeners that have same port on different ip 
stack like ipv4 and ipv6.
+KIP-863: Reduce CompletedFetch#parseRecord() memory 
copyhttps://cwiki.apache.org/confluence/pages/viewpage.action?pageId=225152035;>KIP-863.
 Reduced CompletedFetch#parseRecord() memory copy by deserializing using byte 
buffers.
+KIP-868: Metadata Transactionshttps://cwiki.apache.org/confluence/display/KAFKA/KIP-868+Metadata+Transactions;>KIP-868.
 This feature is about allowing the controller to generate atomic transactions 
of records that can exceed the maximum batch size.
+KIP-902: Upgrade Zookeeper to 3.8.2https://cwiki.apache.org/confluence/display/KAFKA/KIP-902%3A+Upgrade+Zookeeper+to+3.8.1;>KIP-902.
 Zookeeper client upgraded to 3.8.2 version as the current zookeeper dependency 
version 3.6.3 reached the end of life.
+KIP-917: Additional custom metadata for remote log 
segmenthttps://cwiki.apache.org/confluence/display/KAFKA/KIP-917%3A+Additional+custom+metadata+for+remote+log+segment;>KIP-917.
 Custom metedata support for remote log segments.
+KIP-930: Rename ambiguous Tiered Storage Metricshttps://cwiki.apache.org/confluence/display/KAFKA/KIP-930%3A+Rename+ambiguous+Tiered+Storage+Metrics;>KIP-930.
 Renamed ambiguous tiered storage metrics.
+KIP-937: Improve Message Timestamp Validationhttps://cwiki.apache.org/confluence/display/KAFKA/KIP-937%3A+Improve+Message+Timestamp+Validation;>KIP-937.
 Improved Producer's record timestamp validation.
+KIP-938: Add more metrics for measuring KRaft 
performancehttps://cwiki.apache.org/confluence/display/KAFKA/KIP-938%3A+Add+more+metrics+for+measuring+KRaft+performance;>KIP-938.
 Added the targeted KRaft performance metrics mentioned in KIP-938 except 
ForwardingManager metrics.
+
+Kafka Streams
+
+KIP-923: Add A Grace Period to Stream Table Join: 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-923%3A+Add+A+Grace+Period+to+Stream+Table+Join;>KIP-923Added
 a grace period to Stream Table join.
+KIP-941: Range queries to accept null lower and upper 
boundshttps://cwiki.apache.org/confluence/display/KAFKA/KIP-941%3A+Range+queries+to+accept+null+lower+and+upper+bounds;>KIP-941
 Range queries accept lower and upper bounds as null values.
+
+Kafka Connect
+
+KIP-793: Allow sink connectors to be used with 
topic-mutating 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.6 #56

[Apache Jenkins Server]

See