Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #1598

[Apache Jenkins Server]

See 

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1597

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 533179 lines...]
[2023-02-18T00:05:02.051Z] > Task :connect:json:compileTestJava UP-TO-DATE
[2023-02-18T00:05:02.051Z] > Task :raft:compileTestJava UP-TO-DATE
[2023-02-18T00:05:02.051Z] > Task :connect:json:testClasses UP-TO-DATE
[2023-02-18T00:05:02.052Z] > Task :raft:testClasses UP-TO-DATE
[2023-02-18T00:05:02.052Z] > Task :connect:json:testJar
[2023-02-18T00:05:02.052Z] > Task :connect:json:testSrcJar
[2023-02-18T00:05:02.052Z] > Task 
:clients:generateMetadataFileForMavenJavaPublication
[2023-02-18T00:05:02.052Z] > Task :streams:copyDependantLibs
[2023-02-18T00:05:02.052Z] > Task :server-common:compileTestJava UP-TO-DATE
[2023-02-18T00:05:02.052Z] > Task :streams:jar UP-TO-DATE
[2023-02-18T00:05:02.052Z] > Task :server-common:testClasses UP-TO-DATE
[2023-02-18T00:05:02.052Z] > Task :group-coordinator:compileTestJava UP-TO-DATE
[2023-02-18T00:05:02.052Z] > Task :group-coordinator:testClasses UP-TO-DATE
[2023-02-18T00:05:03.007Z] > Task :metadata:compileTestJava UP-TO-DATE
[2023-02-18T00:05:03.007Z] > Task :metadata:testClasses UP-TO-DATE
[2023-02-18T00:05:03.007Z] > Task 
:streams:generateMetadataFileForMavenJavaPublication
[2023-02-18T00:05:04.132Z] 
[2023-02-18T00:05:04.132Z] > Task :connect:api:javadoc
[2023-02-18T00:05:04.132Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk_2/connect/api/src/main/java/org/apache/kafka/connect/source/SourceRecord.java:44:
 warning - Tag @link: reference not found: org.apache.kafka.connect.data
[2023-02-18T00:05:06.087Z] 1 warning
[2023-02-18T00:05:06.087Z] 
[2023-02-18T00:05:06.087Z] > Task :connect:api:copyDependantLibs UP-TO-DATE
[2023-02-18T00:05:06.087Z] > Task :connect:api:jar UP-TO-DATE
[2023-02-18T00:05:06.087Z] > Task 
:connect:api:generateMetadataFileForMavenJavaPublication
[2023-02-18T00:05:06.087Z] > Task :connect:json:copyDependantLibs UP-TO-DATE
[2023-02-18T00:05:06.087Z] > Task :connect:json:jar UP-TO-DATE
[2023-02-18T00:05:06.087Z] > Task 
:connect:json:generateMetadataFileForMavenJavaPublication
[2023-02-18T00:05:07.038Z] > Task 
:connect:json:publishMavenJavaPublicationToMavenLocal
[2023-02-18T00:05:07.038Z] > Task :connect:json:publishToMavenLocal
[2023-02-18T00:05:07.038Z] > Task :connect:api:javadocJar
[2023-02-18T00:05:07.038Z] > Task :connect:api:compileTestJava UP-TO-DATE
[2023-02-18T00:05:07.038Z] > Task :connect:api:testClasses UP-TO-DATE
[2023-02-18T00:05:07.038Z] > Task :connect:api:testJar
[2023-02-18T00:05:07.038Z] > Task :connect:api:testSrcJar
[2023-02-18T00:05:07.038Z] > Task 
:connect:api:publishMavenJavaPublicationToMavenLocal
[2023-02-18T00:05:07.038Z] > Task :connect:api:publishToMavenLocal
[2023-02-18T00:05:10.010Z] > Task :streams:javadoc
[2023-02-18T00:05:10.010Z] > Task :streams:javadocJar
[2023-02-18T00:05:11.758Z] 
[2023-02-18T00:05:11.758Z] > Task :clients:javadoc
[2023-02-18T00:05:11.758Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk_2/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/secured/package-info.java:21:
 warning - Tag @link: reference not found: 
org.apache.kafka.common.security.oauthbearer
[2023-02-18T00:05:11.758Z] 1 warning
[2023-02-18T00:05:12.692Z] 
[2023-02-18T00:05:12.692Z] > Task :clients:javadocJar
[2023-02-18T00:05:13.626Z] > Task :clients:testJar
[2023-02-18T00:05:13.626Z] > Task :clients:testSrcJar
[2023-02-18T00:05:13.626Z] > Task 
:clients:publishMavenJavaPublicationToMavenLocal
[2023-02-18T00:05:13.626Z] > Task :clients:publishToMavenLocal
[2023-02-18T00:05:33.934Z] > Task :core:compileScala
[2023-02-18T00:06:24.401Z] > Task :core:classes
[2023-02-18T00:06:24.401Z] > Task :core:compileTestJava NO-SOURCE
[2023-02-18T00:06:56.062Z] > Task :core:compileTestScala
[2023-02-18T00:07:31.538Z] > Task :core:testClasses
[2023-02-18T00:07:47.773Z] > Task :streams:compileTestJava
[2023-02-18T00:07:47.773Z] > Task :streams:testClasses
[2023-02-18T00:07:47.773Z] > Task :streams:testJar
[2023-02-18T00:07:48.697Z] > Task :streams:testSrcJar
[2023-02-18T00:07:48.697Z] > Task 
:streams:publishMavenJavaPublicationToMavenLocal
[2023-02-18T00:07:48.697Z] > Task :streams:publishToMavenLocal
[2023-02-18T00:07:48.697Z] 
[2023-02-18T00:07:48.697Z] Deprecated Gradle features were used in this build, 
making it incompatible with Gradle 8.0.
[2023-02-18T00:07:48.697Z] 
[2023-02-18T00:07:48.697Z] You can use '--warning-mode all' to show the 
individual deprecation warnings and determine if they come from your own 
scripts or plugins.
[2023-02-18T00:07:48.697Z] 
[2023-02-18T00:07:48.697Z] See 
https://docs.gradle.org/7.6/userguide/command_line_interface.html#sec:command_line_warnings
[2023-02-18T00:07:48.697Z] 
[2023-02-18T00:07:48.697Z] Execution optimizations have been disabled for 2 
invalid unit(s) of work during this build to ensure correctness.
[2023-02-18T00:07:48.697Z] Please consult deprecation warnings for more details.
[2023

Re: [DISCUSS] KIP-858: Handle JBOD broker disk failure in KRaft

[Jun Rao]

Hi, Igor,

Thanks for the reply. A few more replies and comments.

31. Thanks for the explanation. This looks good to me then.

35. Yes, you are right.

36. Yes, this seems fine since the KRaft controller allows broker requests
before it's being unfenced.

37. Yes, it's probably simpler without the optimization for the case with a
single log dir.

41. "If log directory that holds the cluster metadata topic is configured
separately to a different path — using metadata.log.dir — then the
respective UUID for this log directory is excluded from both online and
offline sets, as the broker cannot run if this particular log directory is
unavailable." Hmm, there could be user partitions on metadata.log.dir. So
the controller needs to know the disk UUID for that dir, right? Otherwise,
the controller will keep setting those replicas disk id to zero.

42. "If an entry is removed from log.dirs  the broker can also
automatically update directory.ids as long as no log directories are
offline when the broker comes back up." This makes the removal of a disk
less convenient than before since one can't just remove the disk from
log.dirs and decommission the disk in a single step.

43. "If the broker knows that the partition already exists in a different
log directory that is now offline" The broker may not know that since
currently we remove the offline partitions from LogManager.

44. "When configured for the migration and while still in ZK mode, brokers
will: update meta.properties to generate and include directory.id  and
directory.ids;" I guess there is no downgrade after this point? In KIP-866,
the meta.properties is only changed after the upgrade is finalized.

Thanks,

Jun

On Mon, Feb 6, 2023 at 10:15 AM Igor Soarez 
wrote:

> Hi David,
>
> Thank you for your suggestions and for having a look at this KIP.
>
> 1. Yes, that should be OK. I have updated the section
> "Migrating a cluster in ZK mode running with JBOD" to reflect this.
>
> 2. I've updated the motivation section to state that.
>
> Best,
>
> --
> Igor
>
>
>

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.3 #156

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 501759 lines...]
[2023-02-17T21:40:25.082Z] 
org.apache.kafka.streams.integration.StoreQueryIntegrationTest > 
shouldQuerySpecificStalePartitionStoresMultiStreamThreadsNamedTopology PASSED
[2023-02-17T21:40:25.082Z] 
[2023-02-17T21:40:25.082Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testInnerRepartitioned[caching enabled = true] STARTED
[2023-02-17T21:40:32.127Z] 
[2023-02-17T21:40:32.127Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testInnerRepartitioned[caching enabled = true] PASSED
[2023-02-17T21:40:32.127Z] 
[2023-02-17T21:40:32.127Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testOuterRepartitioned[caching enabled = true] STARTED
[2023-02-17T21:40:40.420Z] 
[2023-02-17T21:40:40.420Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testOuterRepartitioned[caching enabled = true] PASSED
[2023-02-17T21:40:40.420Z] 
[2023-02-17T21:40:40.420Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testInner[caching enabled = true] STARTED
[2023-02-17T21:40:43.364Z] 
[2023-02-17T21:40:43.364Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testInner[caching enabled = true] PASSED
[2023-02-17T21:40:43.364Z] 
[2023-02-17T21:40:43.364Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testOuter[caching enabled = true] STARTED
[2023-02-17T21:40:49.403Z] 
[2023-02-17T21:40:49.403Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testOuter[caching enabled = true] PASSED
[2023-02-17T21:40:49.403Z] 
[2023-02-17T21:40:49.403Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testLeft[caching enabled = true] STARTED
[2023-02-17T21:40:53.418Z] 
[2023-02-17T21:40:53.419Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testLeft[caching enabled = true] PASSED
[2023-02-17T21:40:53.419Z] 
[2023-02-17T21:40:53.419Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testMultiInner[caching enabled = true] STARTED
[2023-02-17T21:41:00.730Z] 
[2023-02-17T21:41:00.730Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testMultiInner[caching enabled = true] PASSED
[2023-02-17T21:41:00.730Z] 
[2023-02-17T21:41:00.730Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testLeftRepartitioned[caching enabled = true] STARTED
[2023-02-17T21:41:07.983Z] 
[2023-02-17T21:41:07.983Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testLeftRepartitioned[caching enabled = true] PASSED
[2023-02-17T21:41:07.983Z] 
[2023-02-17T21:41:07.983Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testSelfJoin[caching enabled = true] STARTED
[2023-02-17T21:41:10.735Z] 
[2023-02-17T21:41:10.735Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testSelfJoin[caching enabled = true] PASSED
[2023-02-17T21:41:10.735Z] 
[2023-02-17T21:41:10.735Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testInnerRepartitioned[caching enabled = false] STARTED
[2023-02-17T21:41:16.576Z] 
[2023-02-17T21:41:16.576Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testInnerRepartitioned[caching enabled = false] PASSED
[2023-02-17T21:41:16.576Z] 
[2023-02-17T21:41:16.576Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testOuterRepartitioned[caching enabled = false] STARTED
[2023-02-17T21:41:25.231Z] 
[2023-02-17T21:41:25.231Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testOuterRepartitioned[caching enabled = false] PASSED
[2023-02-17T21:41:25.231Z] 
[2023-02-17T21:41:25.231Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testInner[caching enabled = false] STARTED
[2023-02-17T21:41:28.757Z] 
[2023-02-17T21:41:28.757Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testInner[caching enabled = false] PASSED
[2023-02-17T21:41:28.757Z] 
[2023-02-17T21:41:28.757Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testOuter[caching enabled = false] STARTED
[2023-02-17T21:41:33.640Z] 
[2023-02-17T21:41:33.640Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testOuter[caching enabled = false] PASSED
[2023-02-17T21:41:33.640Z] 
[2023-02-17T21:41:33.640Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testLeft[caching enabled = false] STARTED
[2023-02-17T21:41:40.948Z] 
[2023-02-17T21:41:40.948Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testLeft[caching enabled = false] PASSED
[2023-02-17T21:41:40.948Z] 
[2023-02-17T21:41:40.948Z] 
org.apache.kafka.streams.integration.Stream

[jira] [Resolved] (KAFKA-13659) MM2 should read all offset syncs at start up

[Chris Egerton (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Egerton resolved KAFKA-13659.
---
Fix Version/s: 3.5.0
   Resolution: Fixed

> MM2 should read all offset syncs at start up
> 
>
> Key: KAFKA-13659
> URL: https://issues.apache.org/jira/browse/KAFKA-13659
> Project: Kafka
>  Issue Type: Improvement
>  Components: mirrormaker
>Reporter: Kanalas Vidor
>Assignee: Greg Harris
>Priority: Major
> Fix For: 3.5.0
>
>
> MirrorCheckpointTask uses OffsetSyncStore, and does not check whether 
> OffsetSyncStore managed to read to the "end" of the offset-syncs topic. 
> OffsetSyncStore should fetch the endoffset of the topic at startup, and set a 
> flag when it finally reaches the endoffset in consumption. 
> MirrorCheckpointTask.poll should wait for this flag to be true before doing 
> any in-memory updates and group offset management.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-12566) Flaky Test MirrorConnectorsIntegrationSSLTest#testReplication

[Chris Egerton (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-12566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Egerton resolved KAFKA-12566.
---
Fix Version/s: 3.5.0
   Resolution: Fixed

> Flaky Test MirrorConnectorsIntegrationSSLTest#testReplication
> -
>
> Key: KAFKA-12566
> URL: https://issues.apache.org/jira/browse/KAFKA-12566
> Project: Kafka
>  Issue Type: Test
>  Components: mirrormaker, unit tests
>Reporter: Matthias J. Sax
>Assignee: Greg Harris
>Priority: Critical
>  Labels: flaky-test
> Fix For: 3.5.0
>
>
>  
> {code:java}
> org.opentest4j.AssertionFailedError: Condition not met within timeout 2. 
> Offsets not translated downstream to primary cluster. ==> expected:  
> but was:  at 
> org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:55) at 
> org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:40) at 
> org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:193) at 
> org.apache.kafka.test.TestUtils.lambda$waitForCondition$3(TestUtils.java:303) 
> at 
> org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:351)
>  at 
> org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:319)
>  at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:300) at 
> org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:290) at 
> org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationBaseTest.testReplication(MirrorConnectorsIntegrationBaseTest.java:289)
> {code}
> {{LOGs}}
> {quote}[2021-03-26 03:28:06,157] ERROR Could not check connector state info. 
> (org.apache.kafka.connect.util.clusters.EmbeddedConnectClusterAssertions:420) 
> org.apache.kafka.connect.runtime.rest.errors.ConnectRestException: Could not 
> read connector state. Error response: \{"error_code":404,"message":"No status 
> found for connector MirrorSourceConnector"} at 
> org.apache.kafka.connect.util.clusters.EmbeddedConnectCluster.connectorStatus(EmbeddedConnectCluster.java:479)
>  at 
> org.apache.kafka.connect.util.clusters.EmbeddedConnectClusterAssertions.checkConnectorState(EmbeddedConnectClusterAssertions.java:413)
>  at 
> org.apache.kafka.connect.util.clusters.EmbeddedConnectClusterAssertions.lambda$assertConnectorAndAtLeastNumTasksAreRunning$16(EmbeddedConnectClusterAssertions.java:286)
>  at 
> org.apache.kafka.test.TestUtils.lambda$waitForCondition$3(TestUtils.java:303) 
> at 
> org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:351)
>  at 
> org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:319)
>  at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:300) at 
> org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:290) at 
> org.apache.kafka.connect.util.clusters.EmbeddedConnectClusterAssertions.assertConnectorAndAtLeastNumTasksAreRunning(EmbeddedConnectClusterAssertions.java:285)
>  at 
> org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationBaseTest.waitUntilMirrorMakerIsRunning(MirrorConnectorsIntegrationBaseTest.java:470)
>  at 
> org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationBaseTest.testReplication(MirrorConnectorsIntegrationBaseTest.java:227){quote}
> and
> {quote}[2021-03-26 03:30:41,524] ERROR [MirrorHeartbeatConnector|task-0] 
> Graceful stop of task MirrorHeartbeatConnector-0 failed. 
> (org.apache.kafka.connect.runtime.Worker:866) [2021-03-26 03:30:41,527] ERROR 
> [MirrorHeartbeatConnector|task-0] 
> WorkerSourceTask\{id=MirrorHeartbeatConnector-0} failed to send record to 
> heartbeats: (org.apache.kafka.connect.runtime.WorkerSourceTask:372) 
> org.apache.kafka.common.KafkaException: Producer is closed forcefully. at 
> org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:750)
>  at 
> org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:737)
>  at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:282) 
> at java.lang.Thread.run(Thread.java:748) [2021-03-26 03:30:42,248] ERROR 
> [MirrorHeartbeatConnector|task-0] 
> WorkerSourceTask\{id=MirrorHeartbeatConnector-0} Failed to flush, timed out 
> while waiting for producer to flush outstanding 1 messages 
> (org.apache.kafka.connect.runtime.WorkerSourceTask:512){quote}
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1596

[Apache Jenkins Server]

See 

MQTT Source Connector support in Open Source Kafka

[Balasubramanian, Anand]

Hi All,

Does the Open Source Kafka version have a MQTT source and sink connector?

OR, is the only option to have that functionality is to use the Confluent Kafka 
version?

Please let me know.

Thanks,
Anand Balasubramanian
Analytics Solutions Architect,
Weatherford.
This electronic communication is for the use of the intended recipient(s) only, 
and may contain confidential, privileged or proprietary information. If you are 
not an intended recipient, please reply to the sender and then immediately 
delete and destroy all copies of the communication. See our Electronic 
Communications Terms and 
Conditions
 for further information

[jira] [Resolved] (KAFKA-14713) Kafka Streams global table startup takes too long

[Matthias J. Sax (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-14713.
-
Resolution: Fixed

> Kafka Streams global table startup takes too long
> -
>
> Key: KAFKA-14713
> URL: https://issues.apache.org/jira/browse/KAFKA-14713
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 3.0.2
>Reporter: Tamas
>Priority: Critical
> Fix For: 3.2.0
>
>
> *Some context first*
> We have a spring based kafka streams application. This application is 
> listening to two topics. Let's call them apartment and visitor. The 
> apartments are stored in a global table, while the visitors are in the stream 
> we are processing, and at one point we are joining the visitor stream 
> together with the apartment table. In our test environment, both topics 
> contain 10 partitions.
> *Issue*
> At first deployment, everything goes fine, the global table is built and all 
> entries in the stream are processed.
> After everything is finished, we shut down the application, restart it and 
> send out a new set of visitors. The application seemingly does not respond.
> After some more debugging it turned out that it simply takes 5 minutes to 
> start up, because the global table takes 30 seconds (default value for the 
> global request timeout) to accept that there are no messages in the apartment 
> topics, for each and every partition. If we send out the list of apartments 
> as new messages, the application starts up immediately.
> To make matters worse, we have clients with 96 partitions, where the startup 
> time would be 48 minutes. Not having messages in the topics between 
> application shutdown and restart is a valid use case, so this is quite a big 
> problem.
> *Possible workarounds*
> We could reduce the request timeout, but since this value is not specific for 
> the global table initialization, but a global request timeout for a lot of 
> things, we do not know what else it will affect, so we are not very keen on 
> doing that. Even then, it would mean a 1.5 minute delay for this particular 
> client (more if we will have other use cases in the future where we will need 
> to use more global tables), which is far too much, considering that the 
> application would be able to otherwise start in about 20 seconds.
> *Potential solutions we see*
>  # Introduce a specific global table initialization timeout in 
> GlobalStateManagerImpl. Then we would be able to safely modify that value 
> without fear of making some other part of kafka unstable.
>  # Parallelize the initialization of the global table partitions in 
> GlobalStateManagerImpl: knowing that the delay at startup is constant instead 
> of linear with the number of partitions would be a huge help.
>  # As long as we receive a response, accept the empty map in the 
> KafkaConsumer, and continue instead of going into a busy-waiting loop.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Reopened] (KAFKA-14713) Kafka Streams global table startup takes too long

[Matthias J. Sax (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax reopened KAFKA-14713:
-

> Kafka Streams global table startup takes too long
> -
>
> Key: KAFKA-14713
> URL: https://issues.apache.org/jira/browse/KAFKA-14713
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 3.0.2
>Reporter: Tamas
>Priority: Critical
> Fix For: 3.2.0
>
>
> *Some context first*
> We have a spring based kafka streams application. This application is 
> listening to two topics. Let's call them apartment and visitor. The 
> apartments are stored in a global table, while the visitors are in the stream 
> we are processing, and at one point we are joining the visitor stream 
> together with the apartment table. In our test environment, both topics 
> contain 10 partitions.
> *Issue*
> At first deployment, everything goes fine, the global table is built and all 
> entries in the stream are processed.
> After everything is finished, we shut down the application, restart it and 
> send out a new set of visitors. The application seemingly does not respond.
> After some more debugging it turned out that it simply takes 5 minutes to 
> start up, because the global table takes 30 seconds (default value for the 
> global request timeout) to accept that there are no messages in the apartment 
> topics, for each and every partition. If we send out the list of apartments 
> as new messages, the application starts up immediately.
> To make matters worse, we have clients with 96 partitions, where the startup 
> time would be 48 minutes. Not having messages in the topics between 
> application shutdown and restart is a valid use case, so this is quite a big 
> problem.
> *Possible workarounds*
> We could reduce the request timeout, but since this value is not specific for 
> the global table initialization, but a global request timeout for a lot of 
> things, we do not know what else it will affect, so we are not very keen on 
> doing that. Even then, it would mean a 1.5 minute delay for this particular 
> client (more if we will have other use cases in the future where we will need 
> to use more global tables), which is far too much, considering that the 
> application would be able to otherwise start in about 20 seconds.
> *Potential solutions we see*
>  # Introduce a specific global table initialization timeout in 
> GlobalStateManagerImpl. Then we would be able to safely modify that value 
> without fear of making some other part of kafka unstable.
>  # Parallelize the initialization of the global table partitions in 
> GlobalStateManagerImpl: knowing that the delay at startup is constant instead 
> of linear with the number of partitions would be a huge help.
>  # As long as we receive a response, accept the empty map in the 
> KafkaConsumer, and continue instead of going into a busy-waiting loop.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-14623) OAuth's HttpAccessTokenRetriever potentially leaks secrets in logging

[Kirk True (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kirk True resolved KAFKA-14623.
---
Resolution: Fixed

> OAuth's HttpAccessTokenRetriever potentially leaks secrets in logging  
> ---
>
> Key: KAFKA-14623
> URL: https://issues.apache.org/jira/browse/KAFKA-14623
> Project: Kafka
>  Issue Type: Bug
>  Components: clients, security
>Affects Versions: 3.3.0, 3.3.1, 3.3.2
>Reporter: Kirk True
>Assignee: Kirk True
>Priority: Major
> Fix For: 3.3.3, 3.4.0
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> The OAuth code that communicates via HTTP with the IdP 
> (HttpAccessTokenRetriever.java) includes logging that outputs the request and 
> response payloads. Among them are:
>  * 
> [https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internals/secured/HttpAccessTokenRetriever.java#L265]
>  * 
> [https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internals/secured/HttpAccessTokenRetriever.java#L274]
>  * 
> [https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internals/secured/HttpAccessTokenRetriever.java#L320]
> It should be determined if there are other places sensitive information might 
> be inadvertently exposed.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Reopened] (KAFKA-14623) OAuth's HttpAccessTokenRetriever potentially leaks secrets in logging

[Kirk True (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kirk True reopened KAFKA-14623:
---

Reopening to fix in 3.1.x and 3.2.x branches.

> OAuth's HttpAccessTokenRetriever potentially leaks secrets in logging  
> ---
>
> Key: KAFKA-14623
> URL: https://issues.apache.org/jira/browse/KAFKA-14623
> Project: Kafka
>  Issue Type: Bug
>  Components: clients, security
>Affects Versions: 3.1.0, 3.2.0, 3.1.1, 3.3.0, 3.1.2, 3.2.1, 3.2.2, 3.2.3, 
> 3.3.1, 3.3.2, 3.2.4
>Reporter: Kirk True
>Assignee: Kirk True
>Priority: Major
> Fix For: 3.4.0, 3.3.3
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> The OAuth code that communicates via HTTP with the IdP 
> (HttpAccessTokenRetriever.java) includes logging that outputs the request and 
> response payloads. Among them are:
>  * 
> [https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internals/secured/HttpAccessTokenRetriever.java#L265]
>  * 
> [https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internals/secured/HttpAccessTokenRetriever.java#L274]
>  * 
> [https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internals/secured/HttpAccessTokenRetriever.java#L320]
> It should be determined if there are other places sensitive information might 
> be inadvertently exposed.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14731) Upgrade ZooKeeper to 3.6.4

[Ron Dagostino (Jira)]

Ron Dagostino created KAFKA-14731:
-

 Summary: Upgrade ZooKeeper to 3.6.4
 Key: KAFKA-14731
 URL: https://issues.apache.org/jira/browse/KAFKA-14731
 Project: Kafka
  Issue Type: Task
Affects Versions: 3.3.2, 3.2.3, 3.4.0, 3.1.2, 3.0.2, 3.5.0
Reporter: Ron Dagostino
Assignee: Ron Dagostino
 Fix For: 3.2.4, 3.1.3, 3.0.3, 3.5.0, 3.4.1, 3.3.3


We have https://issues.apache.org/jira/projects/KAFKA/issues/KAFKA-14661 opened 
to upgrade ZooKeeper from 3.6.3 to 3.8.1, and that will likely be actioned in 
time for 3.5.0.  But in the meantime, ZooKeeper 3.6.4 has been released, so we 
should take the patch version bump in trunk now and also apply the bump to the 
next patch releases of 3.0, 3.1, 3.2, 3.3, and 3.4.

Note that KAFKA-14661 should *not* be applied to branches prior to trunk (and 
presumably 3.5).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-900: KRaft kafka-storage.sh API additions to support SCRAM

[José Armando García Sancio]

Hi Proven,

Thanks for the changes to KIP-900. It looks good to me in general.
Here are some suggestions and questions.

1. In the KIP you give the following example:
--add-scram SCRAM-SHA-256=[user=alice,password=alice-secret]

Is "SCRAM-" required as a prefix? The flag already has the suffix
"-scram". Can the value to the flag be
SHA-256=[user=alice,password=alice-secret]?

2. Should the CLI document all possible values for the --add-scram? Is
SCRAM-SHA-256 the only supported algorithm?

3. Should the short version of the flag --add-scram be -s? I suspect
that in the future we may want to add more options like --add-acl and
--add-config.

Thanks!
-- 
-José

Re: [DISCUSS] KIP-894: Use incrementalAlterConfigs API for syncing topic configurations

[Chris Egerton]

Hi Tina,

This is looking great. I have a few nits remaining but apart from those I'm
happy with the KIP and ready to vote.

1. In the description for how MM2 will behave when configured with
"use.incremental.alter.configs" set to "requested", the KIP states that "If
the first request receives an error from an incompatible broker, it will
fallback to the deprecated AlterConfigs API for the subsequent calls". I
think this should be "If any request receives an error" instead of "If the
first request receives an error" since the first request may fail with a
different error (temporarily unreachable broker, for example), but
subsequent requests may reveal that the targeted cluster does not support
the incremental API.

2. I've realized that I was imagining that
ConfigPropertyFilter::shouldReplicateSourceDefault would accept a single
string parameter (the name of the property to replicate) and return a
boolean value, but this isn't actually laid out in the KIP anywhere. Can
you include a Java snippet of the interface definition for the new method?
It might look something like this if the behavior matches what I had in
mind:

public interface ConfigPropertyFilter extends Configurable, AutoCloseable {
  boolean shouldReplicateSourceDefault(String prop); // New method
}

3. In the "Compatibility, Deprecation, and Migration Plan" section it's
stated that the default value for "use.incremental.alter.configs" will be
"required". I believe this should instead be "requested".

Cheers,

Chris

On Fri, Feb 17, 2023 at 6:26 AM Gantigmaa Selenge 
wrote:

> Hi Chris,
>
> > - The incremental API is used
> - ConfigPropertyFilter::shouldReplicateConfigProperty returns true
> - ConfigPropertyFilter::shouldReplicateSourceDefault returns false
>
> This sounds good to me. So just to clarify this in my head, when
> incremental API is used, the MM2 will check shouldReplicateSourceDefault
> first, which is false by default. When false, as you said it will manually
> delete the default configs in the target cluster. When set to true, it will
> include all the configs including the defaults for syncing.
>
> It would then check shouldReplicateConfigProperty for each config, it will
> return true unless the config is specified in "config.properties.exclude"
> property of DefaultConfigPropertyFilter.
>
> > I also think that extending the DefaultConfigPropertyFilter to allow
> users
> to control which defaults (source or target) get applied on the target
> cluster is worth adding to the KIP. This could be as simple as adding a
> property "use.defaults.from" with accepted values "source" and "target", or
> it could be something more granular like
> "config.properties.source.default.exclude" which, similar to the
> "config.properties.exclude" property, could take a list of regular
> expressions of properties whose default values should not be propagated
> from the source cluster to the target cluster (with a default value of
> ".*", to preserve existing behavior). I'm leaning toward keeping things
> simple for now but both seem like viable options. And of course, if you
> believe we should refrain from doing this, it's at least worth adding to
> the rejected alternatives section.
>
> I agree with extending DefaultConfigPropertyFilter, and I would go with the
> first option that adds  "use.defaults.from". The user can then use the
> existing "config.properties.exclude" property to exclude certain
> configurations from the replication.
>
> I have addressed these now in the KIP.
>
> Regards,
> Tina
>
> On Wed, Feb 15, 2023 at 5:36 PM Chris Egerton 
> wrote:
>
> > Hi Tina,
> >
> > It's looking better! A few thoughts:
> >
> > I think we should clarify in the KIP that under these conditions, MM2
> will
> > explicitly wipe properties from topic configs on the target cluster (via
> a
> > DELETE operation):
> > - The incremental API is used
> > - ConfigPropertyFilter::shouldReplicateConfigProperty returns true
> > - ConfigPropertyFilter::shouldReplicateSourceDefault returns false
> >
> > I also think that extending the DefaultConfigPropertyFilter to allow
> users
> > to control which defaults (source or target) get applied on the target
> > cluster is worth adding to the KIP. This could be as simple as adding a
> > property "use.defaults.from" with accepted values "source" and "target",
> or
> > it could be something more granular like
> > "config.properties.source.default.exclude" which, similar to the
> > "config.properties.exclude" property, could take a list of regular
> > expressions of properties whose default values should not be propagated
> > from the source cluster to the target cluster (with a default value of
> > ".*", to preserve existing behavior). I'm leaning toward keeping things
> > simple for now but both seem like viable options. And of course, if you
> > believe we should refrain from doing this, it's at least worth adding to
> > the rejected alternatives section.
> >
> > Cheers,
> >
> > Chris
> >
> > On Wed, Feb 15, 2023 at 11:19

Re: [ANNOUNCE] New committer: Lucas Bradstreet

[Chris Egerton]

Congrats!

On Fri, Feb 17, 2023 at 9:07 AM Bill Bejeck  wrote:

> Congratulations Lucas!
>
> On Fri, Feb 17, 2023 at 4:28 AM Mickael Maison 
> wrote:
>
> > Congratulations Lucas!
> >
> > On Fri, Feb 17, 2023 at 7:59 AM Tom Bentley  wrote:
> > >
> > > Congratulations!
> > >
> > > On Fri, 17 Feb 2023 at 05:26, Yash Mayya  wrote:
> > >
> > > > Congratulations Lucas!
> > > >
> > > > On Fri, Feb 17, 2023 at 3:25 AM Jun Rao 
> > wrote:
> > > >
> > > > > Hi, Everyone,
> > > > >
> > > > > The PMC of Apache Kafka is pleased to announce a new Kafka
> committer
> > > > Lucas
> > > > > Bradstreet.
> > > > >
> > > > > Lucas has been a long time Kafka contributor since Oct. 2018. He
> has
> > been
> > > > > extremely valuable for Kafka on both performance and correctness
> > > > > improvements.
> > > > >
> > > > > The following are his performance related contributions.
> > > > >
> > > > > KAFKA-9820: validateMessagesAndAssignOffsetsCompressed allocates
> > batch
> > > > > iterator which is not used
> > > > > KAFKA-9685: Solve Set concatenation perf issue in AclAuthorizer
> > > > > KAFKA-9729: avoid readLock in authorizer ACL lookups
> > > > > KAFKA-9039: Optimize ReplicaFetcher fetch path
> > > > > KAFKA-8841: Reduce overhead of
> > ReplicaManager.updateFollowerFetchState
> > > > >
> > > > > The following are his correctness related contributions.
> > > > >
> > > > > KAFKA-13194: LogCleaner may clean past highwatermark
> > > > > KAFKA-10432: LeaderEpochCache is incorrectly recovered on segment
> > > > recovery
> > > > > for epoch 0
> > > > > KAFKA-9137: Fix incorrect FetchSessionCache eviction logic
> > > > >
> > > > > Congratulations, Lucas!
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun (on behalf of the Apache Kafka PMC)
> > > > >
> > > >
> >
>

Re: [ANNOUNCE] New committer: Lucas Bradstreet

[Bill Bejeck]

Congratulations Lucas!

On Fri, Feb 17, 2023 at 4:28 AM Mickael Maison 
wrote:

> Congratulations Lucas!
>
> On Fri, Feb 17, 2023 at 7:59 AM Tom Bentley  wrote:
> >
> > Congratulations!
> >
> > On Fri, 17 Feb 2023 at 05:26, Yash Mayya  wrote:
> >
> > > Congratulations Lucas!
> > >
> > > On Fri, Feb 17, 2023 at 3:25 AM Jun Rao 
> wrote:
> > >
> > > > Hi, Everyone,
> > > >
> > > > The PMC of Apache Kafka is pleased to announce a new Kafka committer
> > > Lucas
> > > > Bradstreet.
> > > >
> > > > Lucas has been a long time Kafka contributor since Oct. 2018. He has
> been
> > > > extremely valuable for Kafka on both performance and correctness
> > > > improvements.
> > > >
> > > > The following are his performance related contributions.
> > > >
> > > > KAFKA-9820: validateMessagesAndAssignOffsetsCompressed allocates
> batch
> > > > iterator which is not used
> > > > KAFKA-9685: Solve Set concatenation perf issue in AclAuthorizer
> > > > KAFKA-9729: avoid readLock in authorizer ACL lookups
> > > > KAFKA-9039: Optimize ReplicaFetcher fetch path
> > > > KAFKA-8841: Reduce overhead of
> ReplicaManager.updateFollowerFetchState
> > > >
> > > > The following are his correctness related contributions.
> > > >
> > > > KAFKA-13194: LogCleaner may clean past highwatermark
> > > > KAFKA-10432: LeaderEpochCache is incorrectly recovered on segment
> > > recovery
> > > > for epoch 0
> > > > KAFKA-9137: Fix incorrect FetchSessionCache eviction logic
> > > >
> > > > Congratulations, Lucas!
> > > >
> > > > Thanks,
> > > >
> > > > Jun (on behalf of the Apache Kafka PMC)
> > > >
> > >
>

Re: [DISCUSS] KIP-905: Broker interceptors

[David Mariassy]

Bumping this thread as I'd love to get a bit more feedback on the general
approach before proceeding.

On Fri, Feb 10, 2023 at 11:41 AM David Mariassy 
wrote:

> Hi Ahmed,
>
> Thanks for taking a look at the KIP, and for your insightful feedback!
>
> I don't disagree with the sentiment that in-band interceptors could be a
> potential source of bugs in a cluster.
>
> Having said that, I don't necessarily think that an in-band interceptor is
> significantly riskier than an out-of-band pre-processor. Let's take the
> example of platform-wide privacy scrubbing. In my opinion it doesn't really
> matter if this feature is deployed as an out-of-band stream processor app
> that consumes from all topics OR if the logic is implemented as an in-ban
> interceptor. Either way, a faulty release of the scrubber will result in
> the platform-wide disruption of data flows. Thus, I'd argue that from the
> perspective of the platform's overall health, the level of risk is very
> comparable in both cases. However in-band interceptors have a couple of
> advantages in my opinion:
> 1. They are significantly cheaper (don't require duplicating data between
> raw and sanitized topics. There are also a lot of potential savings in
> network costs)
> 2. They are easier to maintain (no need to set up additional
> infrastructure for out-of-band processing)
> 3. They can provide accurate produce responses to clients (since there is
> no downstream processing that could render a client's messages invalid
> async)
>
> Also, in-band interceptors could be as safe or risky as their authors
> design them to be. There's nothing stopping someone from catching all
> exceptions in a `processRecord` method, and letting all unprocessed
> messages go through or sending them to a DLQ. Once the interceptor is
> fixed, those unprocessed messages could get re-ingested into Kafka to
> re-attempt pre-processing.
>
> Thanks and happy Friday,
> David
>
>
>
>
>
> On Fri, Feb 10, 2023 at 8:23 AM Ahmed Abdalla 
> wrote:
>
>> Hi David,
>>
>> That's a very interesting KIP and I wanted to share my two cents. I
>> believe
>> there's a lot of value and use cases for the ability to intercept, mutate
>> and filter Kafka's messages, however I'm not sure if trying to achieve
>> that
>> via in-band interceptors is the best approach for this.
>>
>>- My mental model around one of Kafka's core values is the brokers'
>>focus on a single functionality (more or less): highly available and
>> fault
>>tolerant commit log. I see this in many design decisions such as
>>off-loading responsibilities to the clients (partitioner, assignor,
>>consumer groups coordination etc).
>>- And the impact of this KIP on the Kafka server would be adding
>> another
>>moving part to their "state of the world" that they try to maintain.
>> What
>>if an interceptor goes bad? What if there're version-mismatch? etc, a
>> lot
>>of responsibilities that can be managed very efficiently out-of-band
>> IMHO.
>>- The comparison to NginX and Kubernetes is IMHO comparing apples to
>>oranges
>>   - NginX
>>  - Doesn't maintain persisted data.
>>  - It's designed as a middleware, it's an interceptor by nature.
>>   - Kubernetes
>>  - CRDs extend the API surface, they don't "augment" existing
>> APIs.
>>  I think admission webhooks
>>  <
>> https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/
>> >
>> is
>>  Kubernetes' solution for providing interceptors.
>>  - The admission webhooks are out-of-band, and in fact they're a
>>  great example of "opening up your cluster for extensibility"
>> going wrong.
>>  Installing a misbehaving webhook can brick the whole cluster.
>>
>> As I mentioned, I see a value for users being able to intercept and
>> transform Kafka's messages. But I'm worried that having this as a core
>> Kafka feature might not be the best approach for achieving that.
>>
>> Thanks,
>> --
>> Ahmed Abdalla
>> T: @devguyio 
>>
>>
>> On Thu, Feb 9, 2023 at 8:28 PM David Mariassy 
>> wrote:
>>
>> > Hi everyone,
>> >
>> > I'd like to get a discussion going for KIP-905
>> > <
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-905%3A+Broker+interceptors
>> > >,
>> > which proposes the addition of broker interceptors to the stack.
>> >
>> > The KIP contains the motivation, and lists the new public interfaces
>> that
>> > this change would entail. Since my company had its quarterly hack days
>> this
>> > week, I also took the liberty to throw together a first prototype of the
>> > proposed new feature here: https://github.com/apache/kafka/pull/13224.
>> >
>> > Looking forward to the group's feedback!
>> >
>> > Thanks,
>> > David
>> >
>>
>>
>> --
>> *Ahmed Abdalla*
>>
>

[jira] [Created] (KAFKA-14730) Move AdminOperationException to server-commons

[Nikolay Izhikov (Jira)]

Nikolay Izhikov created KAFKA-14730:
---

 Summary: Move AdminOperationException to server-commons
 Key: KAFKA-14730
 URL: https://issues.apache.org/jira/browse/KAFKA-14730
 Project: Kafka
  Issue Type: Sub-task
Reporter: Nikolay Izhikov


AdminOperationException used in `core` module and will be used in `tools` 
module in commands like {{DeleteRecordsCommand}} 

Class need to be moved to `server-commons` module



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-894: Use incrementalAlterConfigs API for syncing topic configurations

[Gantigmaa Selenge]

Hi Chris,

> - The incremental API is used
- ConfigPropertyFilter::shouldReplicateConfigProperty returns true
- ConfigPropertyFilter::shouldReplicateSourceDefault returns false

This sounds good to me. So just to clarify this in my head, when
incremental API is used, the MM2 will check shouldReplicateSourceDefault
first, which is false by default. When false, as you said it will manually
delete the default configs in the target cluster. When set to true, it will
include all the configs including the defaults for syncing.

It would then check shouldReplicateConfigProperty for each config, it will
return true unless the config is specified in "config.properties.exclude"
property of DefaultConfigPropertyFilter.

> I also think that extending the DefaultConfigPropertyFilter to allow users
to control which defaults (source or target) get applied on the target
cluster is worth adding to the KIP. This could be as simple as adding a
property "use.defaults.from" with accepted values "source" and "target", or
it could be something more granular like
"config.properties.source.default.exclude" which, similar to the
"config.properties.exclude" property, could take a list of regular
expressions of properties whose default values should not be propagated
from the source cluster to the target cluster (with a default value of
".*", to preserve existing behavior). I'm leaning toward keeping things
simple for now but both seem like viable options. And of course, if you
believe we should refrain from doing this, it's at least worth adding to
the rejected alternatives section.

I agree with extending DefaultConfigPropertyFilter, and I would go with the
first option that adds  "use.defaults.from". The user can then use the
existing "config.properties.exclude" property to exclude certain
configurations from the replication.

I have addressed these now in the KIP.

Regards,
Tina

On Wed, Feb 15, 2023 at 5:36 PM Chris Egerton 
wrote:

> Hi Tina,
>
> It's looking better! A few thoughts:
>
> I think we should clarify in the KIP that under these conditions, MM2 will
> explicitly wipe properties from topic configs on the target cluster (via a
> DELETE operation):
> - The incremental API is used
> - ConfigPropertyFilter::shouldReplicateConfigProperty returns true
> - ConfigPropertyFilter::shouldReplicateSourceDefault returns false
>
> I also think that extending the DefaultConfigPropertyFilter to allow users
> to control which defaults (source or target) get applied on the target
> cluster is worth adding to the KIP. This could be as simple as adding a
> property "use.defaults.from" with accepted values "source" and "target", or
> it could be something more granular like
> "config.properties.source.default.exclude" which, similar to the
> "config.properties.exclude" property, could take a list of regular
> expressions of properties whose default values should not be propagated
> from the source cluster to the target cluster (with a default value of
> ".*", to preserve existing behavior). I'm leaning toward keeping things
> simple for now but both seem like viable options. And of course, if you
> believe we should refrain from doing this, it's at least worth adding to
> the rejected alternatives section.
>
> Cheers,
>
> Chris
>
> On Wed, Feb 15, 2023 at 11:19 AM Gantigmaa Selenge 
> wrote:
>
> > Thank you Chris. I agree with what you have suggested.
> > I have updated the KIP, please let me know if I missed anything or if
> there
> > is any other question.
> >
> > Regards,
> > Tina
> >
> > On Tue, Feb 14, 2023 at 4:40 PM Chris Egerton 
> > wrote:
> >
> > > Hi Tina,
> > >
> > > While I agree that it's reasonable for users to want to favor the
> source
> > > cluster's defaults over the target cluster's, I'm hesitant to change
> this
> > > behavior in an opt-out fashion. IMO it's better to allow users to opt
> > into
> > > this (by adding a method to the ConfigPropertyFilter interface, and
> > > possibly extending the DefaultConfigPropertyFilter with configuration
> > > properties related to how it should handle source cluster defaults),
> but
> > we
> > > should try to preserve the existing behavior by default.
> > >
> > > Cheers,
> > >
> > > Chris
> > >
> > > On Mon, Feb 13, 2023 at 5:10 PM Gantigmaa Selenge  >
> > > wrote:
> > >
> > > > Hi Chris
> > > >
> > > > My comment on the second point is not correct. Please ignore the part
> > > about
> > > > the config source (config source does set back to DEFAULT_CONFIG when
> > > > deleting a config). I got diverted off the issue a little bit.
> > > >
> > > > With the legacy API, we do propagate deletion due to resetting all
> the
> > > > configs on that target topic that are not being replicated. However
> > with
> > > > incrementalAlterConfigs API, this changes. If we delete a config that
> > was
> > > > previously altered on the source topic, the config on the target
> topic
> > is
> > > > left with the previous value as the default configs are not
> replicated.
> > > The
> > > > reason fo

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1595

[Apache Jenkins Server]

See 

Re: [ANNOUNCE] New committer: Lucas Bradstreet

[Mickael Maison]

Congratulations Lucas!

On Fri, Feb 17, 2023 at 7:59 AM Tom Bentley  wrote:
>
> Congratulations!
>
> On Fri, 17 Feb 2023 at 05:26, Yash Mayya  wrote:
>
> > Congratulations Lucas!
> >
> > On Fri, Feb 17, 2023 at 3:25 AM Jun Rao  wrote:
> >
> > > Hi, Everyone,
> > >
> > > The PMC of Apache Kafka is pleased to announce a new Kafka committer
> > Lucas
> > > Bradstreet.
> > >
> > > Lucas has been a long time Kafka contributor since Oct. 2018. He has been
> > > extremely valuable for Kafka on both performance and correctness
> > > improvements.
> > >
> > > The following are his performance related contributions.
> > >
> > > KAFKA-9820: validateMessagesAndAssignOffsetsCompressed allocates batch
> > > iterator which is not used
> > > KAFKA-9685: Solve Set concatenation perf issue in AclAuthorizer
> > > KAFKA-9729: avoid readLock in authorizer ACL lookups
> > > KAFKA-9039: Optimize ReplicaFetcher fetch path
> > > KAFKA-8841: Reduce overhead of ReplicaManager.updateFollowerFetchState
> > >
> > > The following are his correctness related contributions.
> > >
> > > KAFKA-13194: LogCleaner may clean past highwatermark
> > > KAFKA-10432: LeaderEpochCache is incorrectly recovered on segment
> > recovery
> > > for epoch 0
> > > KAFKA-9137: Fix incorrect FetchSessionCache eviction logic
> > >
> > > Congratulations, Lucas!
> > >
> > > Thanks,
> > >
> > > Jun (on behalf of the Apache Kafka PMC)
> > >
> >