Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.5 #49

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 561204 lines...]
Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
StreamsAssignmentScaleTest > testStickyTaskAssignorLargePartitionCount PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
StreamsAssignmentScaleTest > testFallbackPriorTaskAssignorManyStandbys STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
StreamsAssignmentScaleTest > testFallbackPriorTaskAssignorManyStandbys PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
StreamsAssignmentScaleTest > testHighAvailabilityTaskAssignorManyStandbys 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
StreamsAssignmentScaleTest > testHighAvailabilityTaskAssignorManyStandbys PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
StreamsAssignmentScaleTest > testFallbackPriorTaskAssignorLargeNumConsumers 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
StreamsAssignmentScaleTest > testFallbackPriorTaskAssignorLargeNumConsumers 
PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
StreamsAssignmentScaleTest > testStickyTaskAssignorLargeNumConsumers STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
StreamsAssignmentScaleTest > testStickyTaskAssignorLargeNumConsumers PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
EmitOnChangeIntegrationTest > shouldEmitSameRecordAfterFailover() STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
EmitOnChangeIntegrationTest > shouldEmitSameRecordAfterFailover() PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
HighAvailabilityTaskAssignorIntegrationTest > 
shouldScaleOutWithWarmupTasksAndPersistentStores(TestInfo) STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
HighAvailabilityTaskAssignorIntegrationTest > 
shouldScaleOutWithWarmupTasksAndPersistentStores(TestInfo) PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
HighAvailabilityTaskAssignorIntegrationTest > 
shouldScaleOutWithWarmupTasksAndInMemoryStores(TestInfo) STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
HighAvailabilityTaskAssignorIntegrationTest > 
shouldScaleOutWithWarmupTasksAndInMemoryStores(TestInfo) PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
KStreamAggregationDedupIntegrationTest > shouldReduce(TestInfo) STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
KStreamAggregationDedupIntegrationTest > shouldReduce(TestInfo) PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
KStreamAggregationDedupIntegrationTest > shouldGroupByKey(TestInfo) STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
KStreamAggregationDedupIntegrationTest > shouldGroupByKey(TestInfo) PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
KStreamAggregationDedupIntegrationTest > shouldReduceWindowed(TestInfo) STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
KStreamAggregationDedupIntegrationTest > shouldReduceWindowed(TestInfo) PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
KStreamKStreamIntegrationTest > shouldOuterJoin() STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
KStreamKStreamIntegrationTest > shouldOuterJoin() PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
KTableSourceTopicRestartIntegrationTest > 
shouldRestoreAndProgressWhenTopicNotWrittenToDuringRestoration() STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
KTableSourceTopicRestartIntegrationTest > 
shouldRestoreAndProgressWhenTopicNotWrittenToDuringRestoration() PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
KTableSourceTopicRestartIntegrationTest > 
shouldRestoreAndProgressWhenTopicWrittenToDuringRestorationWithEosAlphaEnabled()
 STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
KTableSourceTopicRestartIntegrationTest > 
shouldRestoreAndProgressWhenTopicWrittenToDuringRestorationWithEosAlphaEnabled()
 PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
KTableSourceTopicRestartIntegrationTest > 
shouldRestoreAndProgressWhenTopicWrittenToDuringRestorationWithEosDisabled() 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
KTableSourceTopicRestartIntegrationTest > 
shouldRestoreAndProgressWhenTopicWrittenToDuringRestorationWithEosDisabled() 
PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 

Re: [DISCUSS] KIP-954: expand default DSL store configuration to custom types

[Almog Gavra]

I have updated the KIP with the points as discussed above. @Guozhang, the
suggested configuration makes it a little more awkward around the
Materialized.as and Materialized.withStoreType APIs than it was when we
were totally deprecating the old configuration. Let me know what you think.

I will open the voting tomorrow! Thanks again everyone for the discussion.

Cheers,
Almog

On Tue, Jul 25, 2023 at 9:20 AM Almog Gavra  wrote:

> Glad you like my KIP-secretary skills ;)
>
> A2. I'm definitely happy to take your suggestion here and not do anything
> special w.r.t. Versioned stores, I think it makes sense especially if we
> consider them implementation details of a specific store type.
>
> At EOD I'll update the KIP with all of these changes and if the
> discussion is silent I'll open a vote tomorrow morning.
>
> Cheers,
> Almog
>
> On Mon, Jul 24, 2023 at 2:02 PM Sophie Blee-Goldman <
> ableegold...@gmail.com> wrote:
>
>> Awesome summary (seriously) -- would you kindly offer your organizational
>> skills to every ongoing KIP from henceforth? We need you :P
>>
>> A few answers/comments:
>>
>> A2: I think there is a 3rd sub-option here, which is to leave
>> versioned-ness out of this KIP entirely, return only the non-versioned
>> stores for now, and then switch over to the versioned stores (only) when
>> the time comes to flip the switch on making them the default across the
>> DSL. This has the advantage of retaining the current behavior/semantics
>> and
>> provides a clear way to transition smoothly in the future, since it seems
>> we will want to cut to all versioned state stores rather than offer users
>> a
>> choice between versioned or non-versioned stores going forward (similar to
>> how we only offer timestamped stores presently, and have completely
>> replaced non-timestamped stores in the DSL.) . In both the timestamped and
>> versioned cases, the old stores are/will still be available or accessible
>> to users via the bare StoreSuppliers, should they somehow desire or
>> require
>> the old store type. Ultimately, I think either this or option (1) would be
>> preferable, though I think it should be up to Matthias or anyone else
>> involved in the versioned stores feature to decide which approach makes
>> sense in the context of that feature's future plans.
>>
>> A3: sounds reasonable to me
>>
>> A5: Also sounds fine to me, though I'll let others chime in with/if they
>> have an alternative suggestion/preference. I guess the other contender
>> would be something like DSLStoreImpl or something like that?
>>
>>
>>
>> On Mon, Jul 24, 2023 at 9:36 AM Almog Gavra 
>> wrote:
>>
>> > Lots of thoughts! Happy to see the thriving discussion on this post -
>> lots
>> > going on so I'm trying to enumerate them to keep things organized
>> (prefix
>> > "A" for "Almog" so we can use numbers in responses for other things ;P).
>> >
>> > A1. Question around closing implementation gaps (e.g. no rocks based
>> > suppression store)
>> > A2. Specifically how to handle Versioned stores
>> > A3. Configuration (new config/reuse old one + new one and ordering of
>> > config resolution)
>> > A4. Drawing a line between what is implementation detail (not exposed in
>> > API) and what is customizable (exposed in API)
>> > A5. Naming of StoreTypeSpec
>> > A6. Param classes in StoreBuilders
>> >
>> > --
>> >
>> > Here are summaries for where it seems each of these stands (trying not
>> to
>> > add any additional opinion yet):
>> >
>> > A1. Sophie/Guozhang/Me (if I count hah!) seem to agree that it is worth
>> > pushing this KIP through independently of the implementation gaps as it
>> > doesn't seem to move the intermediate state further from the end state.
>> > Matthias originally had some concerns.
>> >
>> > A2. There's questions around whether versioned stores should be their
>> own
>> > configurable option or whether they are an implementation detail that
>> the
>> > StoreSpec should decide. It seems like the discussion is converging
>> here,
>> > this should be an implementation detail.
>> >
>> > A3. Matthias/Guozhang prefer adding CUSTOM and then having an additional
>> > config to determine the implementation. Sophie prefers deprecating the
>> old
>> > config. Guozhang additional suggested flipping the resolution order such
>> > that the old config is only respected in a DefaultStoreSpec
>> implementation.
>> >
>> > A4. This KIP (or rather, the discussion on the KIP) blurs the lines
>> between
>> > top level store types (KV, windowed, session) and the underlying
>> > implementation of them (timestamped, versioned, kv-list). It seems
>> everyone
>> > is in alignment to ensure that we keep these two things separate and
>> that
>> > the line is clearly delineated in the text of the KIP.
>> >
>> > A5. Guozhang and Sophie agree that the current name StoreTypeSpec is
>> > misleading, as it's really an implementation spec, not a type
>> > specification.
>> >
>> > A6. Agreement that this is an 

[jira] [Resolved] (KAFKA-15218) NPE will be thrown while deleting topic and fetch from follower concurrently

[Luke Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Chen resolved KAFKA-15218.
---
Fix Version/s: 3.6.0
   Resolution: Fixed

> NPE will be thrown while deleting topic and fetch from follower concurrently
> 
>
> Key: KAFKA-15218
> URL: https://issues.apache.org/jira/browse/KAFKA-15218
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 3.5.0
>Reporter: Luke Chen
>Assignee: Sagar Rao
>Priority: Major
> Fix For: 3.6.0
>
>
> When deleting topics, we'll first clear all the remoteReplicaMap when 
> stopPartitions 
> [here|https://github.com/apache/kafka/blob/2999168cde37142ae3a2377fe939d6b581e692b8/core/src/main/scala/kafka/server/ReplicaManager.scala#L554].
>  But this time, there might be fetch request coming from follower, and try to 
> check if the replica is eligible to be added into ISR 
> [here|https://github.com/apache/kafka/blob/2999168cde37142ae3a2377fe939d6b581e692b8/core/src/main/scala/kafka/cluster/Partition.scala#L1001].
>  At this moment, NPE will be thrown. Although it's fine since this topic is 
> already deleted, it'd be better to avoid it happen.
>  
>  
> {code:java}
> java.lang.NullPointerException: Cannot invoke 
> "kafka.cluster.Replica.stateSnapshot()" because the return value of 
> "kafka.utils.Pool.get(Object)" is null  at 
> kafka.cluster.Partition.isReplicaIsrEligible(Partition.scala:992) 
> ~[kafka_2.13-3.5.0.jar:?]  at 
> kafka.cluster.Partition.canAddReplicaToIsr(Partition.scala:974) 
> ~[kafka_2.13-3.5.0.jar:?]at 
> kafka.cluster.Partition.maybeExpandIsr(Partition.scala:947) 
> ~[kafka_2.13-3.5.0.jar:?]at 
> kafka.cluster.Partition.updateFollowerFetchState(Partition.scala:866) 
> ~[kafka_2.13-3.5.0.jar:?]  at 
> kafka.cluster.Partition.fetchRecords(Partition.scala:1361) 
> ~[kafka_2.13-3.5.0.jar:?] at 
> kafka.server.ReplicaManager.read$1(ReplicaManager.scala:1164) 
> ~[kafka_2.13-3.5.0.jar:?]  at 
> kafka.server.ReplicaManager.$anonfun$readFromLocalLog$7(ReplicaManager.scala:1235)
>  ~[kafka_2.13-3.5.0.jar:?] at 
> scala.collection.IterableOnceOps.foreach(IterableOnce.scala:575) 
> ~[scala-library-2.13.10.jar:?]  at 
> scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:573) 
> ~[scala-library-2.13.10.jar:?] at 
> scala.collection.AbstractIterable.foreach(Iterable.scala:933) 
> ~[scala-library-2.13.10.jar:?] at 
> kafka.server.ReplicaManager.readFromLocalLog(ReplicaManager.scala:1234) 
> ~[kafka_2.13-3.5.0.jar:?]at 
> kafka.server.ReplicaManager.fetchMessages(ReplicaManager.scala:1044) 
> ~[kafka_2.13-3.5.0.jar:?]   at 
> kafka.server.KafkaApis.handleFetchRequest(KafkaApis.scala:994) 
> ~[kafka_2.13-3.5.0.jar:?] at 
> kafka.server.KafkaApis.handle(KafkaApis.scala:181) ~[kafka_2.13-3.5.0.jar:?] 
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:76) 
> ~[kafka_2.13-3.5.0.jar:?] at java.lang.Thread.run(Thread.java:1623) [?:?] 
> {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15251) Upgrade system test to use 3.5.1

[Matthias J. Sax (Jira)]

Matthias J. Sax created KAFKA-15251:
---

 Summary: Upgrade system test to use 3.5.1
 Key: KAFKA-15251
 URL: https://issues.apache.org/jira/browse/KAFKA-15251
 Project: Kafka
  Issue Type: Test
  Components: streams, system tests
Reporter: Matthias J. Sax


3.5.1 was released and we should update the upgrade system tests accordingly to 
use the new version



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-955: Add stream-table join on foreign key

[Matthias J. Sax]

Igor,

thanks for the KIP. Interesting proposal. I am wondering a little bit 
about the use-case and semantics, and if it's really required to add 
what you propose? Please correct me if I am wrong.


In the end, a stream-table join is a "stream enrichment" (via a table 
lookup). Thus, it's inherently a 1:1 join (in contrast to a FK 
table-table join which is a n:1 join).


If this assumption is correct, and you have data for which the table 
side join attribute is in the value, you could actually repartition the 
table data using the join attribute as the PK of the table.


If my assumption is incorrect, and you say you want to have a 1:n join 
(note that I intentionally reversed from n:1 to 1:n), I would rather 
object, because it seems to violate the idea to "enrich" a stream, what 
means that each input record produced an output record, not multiple?


Also note: for a FK table-table join, we use the forgeinKeyExtractor to 
get the join attribute from the left input table (which corresponds to 
the KStream in your KIP; ie, it's a n:1 join), while you propose to use 
the foreignKeyExtractor to be applied to the KTable (which is the right 
input, and thus it would be a 1:n join).


Maybe you can clarify the use case a little bit. For the current KIP 
description I only see the 1:1 join case, what would mean we might not 
need such a feature?



-Matthias


On 7/24/23 11:36 AM, Igor Fomenko wrote:

Hello developers of the Kafka Streams,

I would like to start discussion on KIP-955: Add stream-table join on
foreign key

This KIP proposes the new API to join KStrem with KTable based on foreign
key relation.
Ths KIP was inspired by one of my former projects to integrate RDBMS
databases with data consumers using Change Data Capture and Kafka.
If we had the capability in Kafka Stream to join KStream with KTable on
foreign key this would simplify our implementation significantly.

Looking forward to your feedback and discussion.

Regards,

Igor

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.5 #48

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 474100 lines...]
> Task :connect:json:generateMetadataFileForMavenJavaPublication
> Task :connect:api:javadocJar
> Task :connect:api:compileTestJava UP-TO-DATE
> Task :connect:api:testClasses UP-TO-DATE
> Task :connect:api:testJar
> Task :connect:api:testSrcJar
> Task :connect:json:publishMavenJavaPublicationToMavenLocal
> Task :connect:api:publishMavenJavaPublicationToMavenLocal
> Task :connect:json:publishToMavenLocal
> Task :connect:api:publishToMavenLocal
> Task :streams:javadoc
> Task :streams:javadocJar

> Task :clients:javadoc
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.5/clients/src/main/java/org/apache/kafka/clients/admin/ScramMechanism.java:32:
 warning - Tag @see: missing final '>': "https://cwiki.apache.org/confluence/display/KAFKA/KIP-554%3A+Add+Broker-side+SCRAM+Config+API;>KIP-554:
 Add Broker-side SCRAM Config API

 This code is duplicated in 
org.apache.kafka.common.security.scram.internals.ScramMechanism.
 The type field in both files must match and must not change. The type field
 is used both for passing ScramCredentialUpsertion and for the internal
 UserScramCredentialRecord. Do not change the type field."
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.5/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/secured/package-info.java:21:
 warning - Tag @link: reference not found: 
org.apache.kafka.common.security.oauthbearer
2 warnings

> Task :clients:javadocJar
> Task :clients:srcJar
> Task :clients:testJar
> Task :clients:testSrcJar
> Task :clients:publishMavenJavaPublicationToMavenLocal
> Task :clients:publishToMavenLocal
> Task :core:compileScala
> Task :core:classes
> Task :core:compileTestJava NO-SOURCE
> Task :core:compileTestScala
> Task :core:testClasses
> Task :streams:compileTestJava
> Task :streams:testClasses
> Task :streams:testJar
> Task :streams:testSrcJar
> Task :streams:publishMavenJavaPublicationToMavenLocal
> Task :streams:publishToMavenLocal

Deprecated Gradle features were used in this build, making it incompatible with 
Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings 
and determine if they come from your own scripts or plugins.

See 
https://docs.gradle.org/8.0.2/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 3m 36s
89 actionable tasks: 35 executed, 54 up-to-date
[Pipeline] sh
+ grep ^version= gradle.properties
+ cut -d= -f 2
[Pipeline] dir
Running in 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.5/streams/quickstart
[Pipeline] {
[Pipeline] sh
+ mvn clean install -Dgpg.skip
[INFO] Scanning for projects...
[INFO] 
[INFO] Reactor Build Order:
[INFO] 
[INFO] Kafka Streams :: Quickstart[pom]
[INFO] streams-quickstart-java[maven-archetype]
[INFO] 
[INFO] < org.apache.kafka:streams-quickstart >-
[INFO] Building Kafka Streams :: Quickstart 3.5.2-SNAPSHOT[1/2]
[INFO]   from pom.xml
[INFO] [ pom ]-
[INFO] 
[INFO] --- clean:3.0.0:clean (default-clean) @ streams-quickstart ---
[INFO] 
[INFO] --- remote-resources:1.5:process (process-resource-bundles) @ 
streams-quickstart ---
[INFO] 
[INFO] --- site:3.5.1:attach-descriptor (attach-descriptor) @ 
streams-quickstart ---
[INFO] 
[INFO] --- gpg:1.6:sign (sign-artifacts) @ streams-quickstart ---
[INFO] 
[INFO] --- install:2.5.2:install (default-install) @ streams-quickstart ---
[INFO] Installing 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.5/streams/quickstart/pom.xml
 to 
/home/jenkins/.m2/repository/org/apache/kafka/streams-quickstart/3.5.2-SNAPSHOT/streams-quickstart-3.5.2-SNAPSHOT.pom
[INFO] 
[INFO] --< org.apache.kafka:streams-quickstart-java >--
[INFO] Building streams-quickstart-java 3.5.2-SNAPSHOT[2/2]
[INFO]   from java/pom.xml
[INFO] --[ maven-archetype ]---
[INFO] 
[INFO] --- clean:3.0.0:clean (default-clean) @ streams-quickstart-java ---
[INFO] 
[INFO] --- remote-resources:1.5:process (process-resource-bundles) @ 
streams-quickstart-java ---
[INFO] 
[INFO] --- resources:2.7:resources (default-resources) @ 
streams-quickstart-java ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 6 resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- resources:2.7:testResources (default-testResources) @ 
streams-quickstart-java ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- archetype:2.2:jar (default-jar) @ streams-quickstart-java ---
[INFO] Building archetype jar: 

Re: [VOTE] KIP-858: Handle JBOD broker disk failure in KRaft

[Igor Soarez]

Hi Ismael,

I believe I have addressed all concerns.
Please have a look, and consider a vote on this KIP.

Thank you,

--
Igor

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2037

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 295536 lines...]

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testInnerOuter[caching enabled = false] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testInnerLeft[caching enabled = false] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testInnerLeft[caching enabled = false] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testOuterInner[caching enabled = false] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testOuterInner[caching enabled = false] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testOuterOuter[caching enabled = false] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testOuterOuter[caching enabled = false] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testInnerWithRightVersionedOnly[caching enabled = false] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testInnerWithRightVersionedOnly[caching enabled = false] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testLeftWithLeftVersionedOnly[caching enabled = false] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testLeftWithLeftVersionedOnly[caching enabled = false] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testInnerWithLeftVersionedOnly[caching enabled = false] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testInnerWithLeftVersionedOnly[caching enabled = false] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
TaskAssignorIntegrationTest > shouldProperlyConfigureTheAssignor STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
TaskAssignorIntegrationTest > shouldProperlyConfigureTheAssignor PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
TaskMetadataIntegrationTest > shouldReportCorrectEndOffsetInformation STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
TaskMetadataIntegrationTest > shouldReportCorrectEndOffsetInformation PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
TaskMetadataIntegrationTest > shouldReportCorrectCommittedOffsetInformation 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
TaskMetadataIntegrationTest > shouldReportCorrectCommittedOffsetInformation 
PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
EmitOnChangeIntegrationTest > shouldEmitSameRecordAfterFailover() STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
EmitOnChangeIntegrationTest > shouldEmitSameRecordAfterFailover() PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
HighAvailabilityTaskAssignorIntegrationTest > 
shouldScaleOutWithWarmupTasksAndPersistentStores(TestInfo) STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
HighAvailabilityTaskAssignorIntegrationTest > 
shouldScaleOutWithWarmupTasksAndPersistentStores(TestInfo) PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
HighAvailabilityTaskAssignorIntegrationTest > 
shouldScaleOutWithWarmupTasksAndInMemoryStores(TestInfo) STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
HighAvailabilityTaskAssignorIntegrationTest > 
shouldScaleOutWithWarmupTasksAndInMemoryStores(TestInfo) PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
KStreamAggregationDedupIntegrationTest > shouldReduce(TestInfo) STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
KStreamAggregationDedupIntegrationTest > shouldReduce(TestInfo) PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
KStreamAggregationDedupIntegrationTest > shouldGroupByKey(TestInfo) STARTED

Gradle Test Run 

Re: [VOTE] KIP-949: Add flag to enable the usage of topic separator in MM2 DefaultReplicationPolicy

[Greg Harris]

Hey Omnia,

Thanks for the KIP!

I think that MM2 is responsible for providing an upgrade path for
users, even if it isn't backwards-compatible by default due to a
mistake.
The non-configuration-based strategies I could think of aren't viable
due to the danger of inferring the incorrect topic name, and inherent
complexity which makes them hard to backport.
I also support the decision to backport this to 3.1 - 3.5, so that MM2
users can upgrade in minor version increments after those patch
releases go out.

I'm +1 (binding).

Thanks,
Greg

On Mon, Jul 24, 2023 at 7:21 AM Omnia Ibrahim  wrote:
>
> Hi Chris, I updated the KIP to address your feedback. Thanks for the vote.
>
> On Mon, Jul 24, 2023 at 1:30 PM Chris Egerton 
> wrote:
>
> > Hi Omnia,
> >
> > I think there's a few clarifications that should still be made on the KIP,
> > but assuming these are agreeable, I'm +1 (binding)
> >
> > - In the description for the
> > replication.policy.internal.topic.separator.enabled property (in the
> > "Public Interfaces" section), we should specify that it affects only the
> > checkpoints and offset syncs topic
> > - We can remove the code snippet from the "Proposed Changes" section (right
> > now it's a little buggy; there's two different implementations for the same
> > "internalSuffix" method, and there are references to an "internalSeparator"
> > method but no implementation for it); since we don't usually require
> > specific code changes in KIPs, I think as long as we can describe the
> > changes we're proposing in the "Public Interfaces" section, that should be
> > enough for this KIP
> >
> > Cheers,
> >
> > Chris
> >
> > On Mon, Jul 24, 2023 at 2:04 AM Federico Valeri 
> > wrote:
> >
> > > +1 (non binding)
> > >
> > > Thanks
> > > Fede
> > >
> > >
> > > On Sun, Jul 23, 2023 at 6:30 PM Omnia Ibrahim 
> > > wrote:
> > > >
> > > > Hi everyone,
> > > > I would like to open a vote for KIP-949. The proposal is here
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-949%3A+Add+flag+to+enable+the+usage+of+topic+separator+in+MM2+DefaultReplicationPolicy
> > > > .
> > > > <
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-949%3A+Add+flag+to+enable+the+usage+of+topic+separator+in+MM2+DefaultReplicationPolicy
> > > >
> > > >
> > > > Thanks
> > > > Omnia
> > >
> >

[jira] [Created] (KAFKA-15250) DefaultBackgroundThread is running tight loop

[Philip Nee (Jira)]

Philip Nee created KAFKA-15250:
--

 Summary: DefaultBackgroundThread is running tight loop
 Key: KAFKA-15250
 URL: https://issues.apache.org/jira/browse/KAFKA-15250
 Project: Kafka
  Issue Type: Bug
  Components: consumer
Reporter: Philip Nee
Assignee: Philip Nee


The DefaultBackgroundThread is running tight loops and wasting CPU cycles.  I 
think we need to reexamine the timeout pass to networkclientDelegate.poll.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: KIP-919: Allow AdminClient to Talk Directly with the KRaft Controller Quorum

[Colin McCabe]

On Tue, Jul 25, 2023, at 05:30, Luke Chen wrote:
> Hi Colin,
>
> Some more comments:
> 1. In the KIP, we mentioned "controller heartbeats", but it is not
> explained anywhere.
> I think "controller heartbeats" = controller registration", is that
> correct?
> If no, please explain more about it in the KIP.

Hi Luke,

Sorry, this was an artifact of earlier revisions. I have changed it to 
"ControllerRegistration" in all the cases where I didn't update it before.

>
> 2. Following this question:
> > Which endpoint will the inactive controllers use to send the
> > ControllerRegistrationRequest?
> > A: They will use the endpoint in controller.quorum.voters.
> If the registration request will include controller.quorum.voters, why
> bother sending this information to active controller again?
> The active controller should already have all the controller.quorum.voters
> when start up.
> Any purpose of that design? For validation?

The controllers don't know which endpoint controller.quorum.voters is 
referencing.

>
> 3. If a controller node is not part of `controller.quorum.voters`, when it
> sends ControllerRegistrationRequest, what will we respond to it?
>

Good point. I added a new error, UNKNOWN_CONTROLLER_ID, for this case.

> 4. Nice and clear compatibility matrix!
>

Thanks!
Colin

> Thank you.
> Luke
>
> On Sat, Jul 22, 2023 at 3:38 AM Colin McCabe  wrote:
>
>> On Fri, Jul 21, 2023, at 09:43, José Armando García Sancio wrote:
>> > Thanks for the KIP Colin. Apologies if some of these points have
>> > already been made. I have not followed the discussion closely:
>> >
>> > 1. Re: Periodically, each controller will check that the controller
>> > registration for its ID is as expected
>> >
>> > Does this need to be periodic? Can't the controller schedule this RPC,
>> > retry etc, when it finds that the incarnation ID doesn't match its
>> > own?
>> >
>>
>> Hi José,
>>
>> Thanks for the reviews.
>>
>> David had the same question. I agree that it should be event-driven rather
>> than periodic (except for retries, etc.)
>>
>> >
>> > 2. Did you consider including the active controller's epoch in the
>> > ControllerRegistrationRequest?
>> >
>> > This would allow the active controller to reject registration from
>> > controllers that are not part of the active quorum and don't know the
>> > latest controller epoch. This should mitigate some of the concerns you
>> > raised in bullet point 1.
>> >
>>
>> Good idea. I will add the active controller epoch to the registration
>> request.
>>
>> >
>> > 3. Which endpoint will the inactive controllers use to send the
>> > ControllerRegistrationRequest?
>> >
>> > Will it use the first endpoint described in the cluster metadata
>> > controller registration record? Or would it use the endpoint described
>> > in the server configuration at controller.quorum.voters?
>> >
>>
>> They will use the endpoint in controller.quorum.voters. In general, the
>> endpoints from the registration are only used for responding to
>> DESCRIBE_CLUSTER. Since, after all, we may not even have the registration
>> endpoints when we start up.
>>
>> >
>> > 4. Re: Raft integration in the rejected alternatives
>> >
>> > Yes, The KRaft layer needs to solve a similar problem like endpoint
>> > discovery to support dynamic controller membership change. As you
>> > point out the requirements are different and the set of information
>> > that needs to be tracked is different. I think it is okay to use a
>> > different solution for each of these problems.
>>
>> Yeah that was my feeling too. Thanks for taking a look.
>>
>> regards,
>> Colin
>>
>> >
>> > Thanks!
>> > --
>> > -José
>>

Debugging Jenkins test failures

[Kirk True]

Hi all!

I’ve noticed that we’re back in the state where it’s tough to get a clean PR 
Jenkins test run. Spot checking the top ~10 pull request runs show this doesn’t 
appear to be an issue with just my PRs :P

I know we have some chronic flaky tests, but I’ve seen at least two other 
classes of problems:

1. Jenkins test runners hanging and eventually timing out
2. Intra Jenkins-container/pod/VM/machine/turtle communication issues

How do we go about diagnosing test runs that fail in such an opaque fashion?

Thanks!
Kirk

Re: Increase in flaky test failures

[Kirk True]

Hi Divij,

I’m hitting something similar in my pull request test run 
(https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-13990/26/pipeline/10):

> Task :clients:testClasses
Unexpected exception thrown.
org.gradle.internal.remote.internal.MessageIOException: Could not read message 
from '/127.0.0.1:53070'.
at 
org.gradle.internal.remote.internal.inet.SocketConnection.receive(SocketConnection.java:94)
at 
org.gradle.internal.remote.internal.hub.MessageHub$ConnectionReceive.run(MessageHub.java:270)
at 
org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
at 
org.gradle.internal.concurrent.AbstractManagedExecutor$1.run(AbstractManagedExecutor.java:47)
at 
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at 
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.IllegalArgumentException
at 
org.gradle.internal.remote.internal.hub.InterHubMessageSerializer$MessageReader.read(InterHubMessageSerializer.java:72)
at 
org.gradle.internal.remote.internal.hub.InterHubMessageSerializer$MessageReader.read(InterHubMessageSerializer.java:52)
at 
org.gradle.internal.remote.internal.inet.SocketConnection.receive(SocketConnection.java:81)
... 6 more
> Task :clients:spotbugsTest SKIPPED
> Task :connect:mirror-client:compileTestJava
> Task :connect:mirror-client:testClasses
> Task :storage:api:compileTestJava
> Task :storage:api:testClasses
> Task :log4j-appender:compileTestJava
> Task :log4j-appender:testClasses
org.gradle.internal.remote.internal.ConnectException: Could not connect to 
server [871f8055-03e7-4c5c-bc92-6b367d765631 port:37149, 
addresses:[/127.0.0.1]]. Tried addresses: [/127.0.0.1].
at 
org.gradle.internal.remote.internal.inet.TcpOutgoingConnector.connect(TcpOutgoingConnector.java:67)
at 
org.gradle.internal.remote.internal.hub.MessageHubBackedClient.getConnection(MessageHubBackedClient.java:36)
at 
org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:103)
at 
org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:65)
at 
worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
at 
worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)
Caused by: java.net.ConnectException: Connection refused
at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at 
java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:774)
at java.base/sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:120)
at 
org.gradle.internal.remote.internal.inet.TcpOutgoingConnector.tryConnect(TcpOutgoingConnector.java:81)
at 
org.gradle.internal.remote.internal.inet.TcpOutgoingConnector.connect(TcpOutgoingConnector.java:54)
... 5 more
> Task :streams:examples:compileTestJava
> Task :connect:mirror-client:spotbugsTest SKIPPED
> Task :storage:api:spotbugsTest SKIPPED
> Task :streams:examples:testClasses
> Task :clients:checkstyleTest FAILED

This only pops up occasionally, of course :(

If you got the chance to look into this before, did you find anything promising 
to investigate/research?

Thanks,
Kirk

> On Jun 15, 2023, at 11:32 AM, Divij Vaidya  wrote:
> 
> Hey folks
> 
> We are having increased flaky failures for the past one day.
> 
> Mostly, streams upgrade tests (:streams:upgrade-system-tests-20:unitTest)
> are failing with
> 
> org.gradle.internal.remote.internal.ConnectException: Could not
> connect to server [90d54b62-a664-438e-b241-20847ead1eab port:34765,
> addresses:[/127.0.0.1]]. Tried addresses: [/127.0.0.1].
> [2023-06-15T14:50:31.779Z]at
> org.gradle.internal.remote.internal.inet.TcpOutgoingConnector.connect(TcpOutgoingConnector.java:67)
> 
> 
> And other network related tests are failing with errors such as
> 
> java.net.ConnectException: Connection timed out
> at java.base/sun.nio.ch.Net.connect0(Native Method)
> at java.base/sun.nio.ch.Net.connect(Net.java:579)
> at java.base/sun.nio.ch.Net.connect(Net.java:568)
> at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:588)
> at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
> at java.base/java.net.Socket.connect(Socket.java:633)
> at java.base/java.net.Socket.connect(Socket.java:583)
> at java.base/java.net.Socket.(Socket.java:507)
> at java.base/java.net.Socket.(Socket.java:319)
> at
> org.apache.kafka.common.network.PlaintextSender.lambda$new$0(PlaintextSender.java:30)
> 
> 
> These stack traces probably hints at some infra problem to me? I will take
> a deeper look tomorrow in my day time but meanwhile, 

Re: [VOTE] KIP-930: Tiered Storage Metrics

[Divij Vaidya]

Thank you for the KIP Abhinav. Although we should avoid changing
customer-facing interfaces (such as metrics) after a KIP is accepted,
in this case, I think that the divergence is minimal and the right
thing to do in the longer run. Hence, I would consider this change as
a one-off exception and not a precedent for the future changes.

+1 (binding) from me.

Also, I think we should leave the vote open longer for some duration
(at least 2 weeks) to give an opportunity for folks in the community
to add any thoughts that they might have. The KIP has been published
for only 1 day so far and interested folks may not have had a chance
to look into it yet.

--
Divij Vaidya



On Tue, Jul 25, 2023 at 6:45 PM Satish Duggana  wrote:
>
> +1 for the KIP.
>
> Thanks,
> Satish.
>
> On Tue, 25 Jul 2023 at 18:31, Kamal Chandraprakash
>  wrote:
> >
> > +1 (non-binding)
> >
> > --
> > Kamal
> >
> > On Tue, Jul 25, 2023 at 11:30 AM Abhijeet Kumar 
> > wrote:
> >
> > > Hi All,
> > >
> > > I would like to start the vote for KIP-930 Tiered Storage Metrics.
> > >
> > > The KIP is here:
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-930%3A+Tiered+Storage+Metrics
> > >
> > > Regards
> > > Abhijeet.
> > >

Re: [VOTE] KIP-959 Add BooleanConverter to Kafka Connect

[Yash Mayya]

Hi Hector,

Thanks for the KIP!

+1 (non-binding)

Thanks,
Yash

On Tue, Jul 25, 2023 at 11:01 PM Andrew Schofield <
andrew_schofield_j...@outlook.com> wrote:

> Thanks for the KIP. As you say, not that controversial.
>
> +1 (non-binding)
>
> Thanks,
> Andrew
>
> > On 25 Jul 2023, at 18:22, Hector Geraldino (BLOOMBERG/ 919 3RD A) <
> hgerald...@bloomberg.net> wrote:
> >
> > Hi everyone,
> >
> > The changes proposed by KIP-959 (Add BooleanConverter to Kafka Connect)
> have a limited scope and shouldn't be controversial. I'm opening a voting
> thread with the hope that it can be included in the next upcoming 3.6
> release.
> >
> > Here are some links:
> >
> > KIP:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-959%3A+Add+BooleanConverter+to+Kafka+Connect
> > JIRA: https://issues.apache.org/jira/browse/KAFKA-15248
> > Discussion thread:
> https://lists.apache.org/thread/15c2t0kl9bozmzjxmkl5n57kv4l4o1dt
> > Pull Request: https://github.com/apache/kafka/pull/14093
> >
> > Thanks!
>
>
>

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2036

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 296992 lines...]
Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testInnerInner[caching enabled = false] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testInnerOuter[caching enabled = false] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testInnerOuter[caching enabled = false] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testInnerLeft[caching enabled = false] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testInnerLeft[caching enabled = false] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testOuterInner[caching enabled = false] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testOuterInner[caching enabled = false] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testOuterOuter[caching enabled = false] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testOuterOuter[caching enabled = false] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testInnerWithRightVersionedOnly[caching enabled = false] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testInnerWithRightVersionedOnly[caching enabled = false] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testLeftWithLeftVersionedOnly[caching enabled = false] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testLeftWithLeftVersionedOnly[caching enabled = false] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testInnerWithLeftVersionedOnly[caching enabled = false] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testInnerWithLeftVersionedOnly[caching enabled = false] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
TaskAssignorIntegrationTest > shouldProperlyConfigureTheAssignor STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
TaskAssignorIntegrationTest > shouldProperlyConfigureTheAssignor PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
TaskMetadataIntegrationTest > shouldReportCorrectEndOffsetInformation STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
TaskMetadataIntegrationTest > shouldReportCorrectEndOffsetInformation PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
TaskMetadataIntegrationTest > shouldReportCorrectCommittedOffsetInformation 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
TaskMetadataIntegrationTest > shouldReportCorrectCommittedOffsetInformation 
PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
EmitOnChangeIntegrationTest > shouldEmitSameRecordAfterFailover() STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
EmitOnChangeIntegrationTest > shouldEmitSameRecordAfterFailover() PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
HighAvailabilityTaskAssignorIntegrationTest > 
shouldScaleOutWithWarmupTasksAndPersistentStores(TestInfo) STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
HighAvailabilityTaskAssignorIntegrationTest > 
shouldScaleOutWithWarmupTasksAndPersistentStores(TestInfo) PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
HighAvailabilityTaskAssignorIntegrationTest > 
shouldScaleOutWithWarmupTasksAndInMemoryStores(TestInfo) STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
HighAvailabilityTaskAssignorIntegrationTest > 
shouldScaleOutWithWarmupTasksAndInMemoryStores(TestInfo) PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 

Re: [VOTE] KIP-858: Handle JBOD broker disk failure in KRaft

[Ismael Juma]

Thanks Igor. Are there unaddressed concerns or is this ready for a vote
again?

Ismael

On Tue, Jul 25, 2023 at 6:14 PM Igor Soarez 
wrote:

> Hi everyone,
>
> Following a face-to-face discussion with Ron and Colin,
> I have just made further improvements to this KIP:
>
>
> 1. Every log directory gets a random UUID assigned, even if just one
>log dir is configured in the Broker.
>
> 2. All online log directories are registered, even if just one if
> configured.
>
> 3. Partition-to-directory assignments are only performed if more than
>one log directory is configured/registered.
>
> 4. A successful reply from the Controller to a AssignReplicasToDirsRequest
>is taken as a guarantee that the metadata changes are
>successfully persisted.
>
> 5. Replica assignments that refer log directories pending a failure
>notification are prioritized to guarantee the Controller and Broker
>agree on the assignments before acting on the failure notification.
>
> 6. The transition from one log directory to multiple log directories
>relies on a logical update to efficiently update directory assignments
>to the previously registered single log directory when that's possible.
>
> I have also introduced a configuration for the maximum time the broker
> will keep trying to send a log directory notification before shutting down.
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-858%3A+Handle+JBOD+broker+disk+failure+in+KRaft
>
> Best,
>
> --
> Igor
>
>

Re: [VOTE] KIP-959 Add BooleanConverter to Kafka Connect

[Andrew Schofield]

Thanks for the KIP. As you say, not that controversial.

+1 (non-binding)

Thanks,
Andrew

> On 25 Jul 2023, at 18:22, Hector Geraldino (BLOOMBERG/ 919 3RD A) 
>  wrote:
>
> Hi everyone,
>
> The changes proposed by KIP-959 (Add BooleanConverter to Kafka Connect) have 
> a limited scope and shouldn't be controversial. I'm opening a voting thread 
> with the hope that it can be included in the next upcoming 3.6 release.
>
> Here are some links:
>
> KIP: 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-959%3A+Add+BooleanConverter+to+Kafka+Connect
> JIRA: https://issues.apache.org/jira/browse/KAFKA-15248
> Discussion thread: 
> https://lists.apache.org/thread/15c2t0kl9bozmzjxmkl5n57kv4l4o1dt
> Pull Request: https://github.com/apache/kafka/pull/14093
>
> Thanks!

[VOTE] KIP-959 Add BooleanConverter to Kafka Connect

[Hector Geraldino (BLOOMBERG/ 919 3RD A)]

Hi everyone,

The changes proposed by KIP-959 (Add BooleanConverter to Kafka Connect) have a 
limited scope and shouldn't be controversial. I'm opening a voting thread with 
the hope that it can be included in the next upcoming 3.6 release. 

Here are some links:

KIP: 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-959%3A+Add+BooleanConverter+to+Kafka+Connect
JIRA: https://issues.apache.org/jira/browse/KAFKA-15248
Discussion thread: 
https://lists.apache.org/thread/15c2t0kl9bozmzjxmkl5n57kv4l4o1dt
Pull Request: https://github.com/apache/kafka/pull/14093

Thanks!

Re: [VOTE] KIP-858: Handle JBOD broker disk failure in KRaft

[Igor Soarez]

Hi everyone,

Following a face-to-face discussion with Ron and Colin,
I have just made further improvements to this KIP:


1. Every log directory gets a random UUID assigned, even if just one
   log dir is configured in the Broker.

2. All online log directories are registered, even if just one if configured.

3. Partition-to-directory assignments are only performed if more than
   one log directory is configured/registered.

4. A successful reply from the Controller to a AssignReplicasToDirsRequest
   is taken as a guarantee that the metadata changes are
   successfully persisted.

5. Replica assignments that refer log directories pending a failure
   notification are prioritized to guarantee the Controller and Broker
   agree on the assignments before acting on the failure notification.

6. The transition from one log directory to multiple log directories
   relies on a logical update to efficiently update directory assignments
   to the previously registered single log directory when that's possible.

I have also introduced a configuration for the maximum time the broker
will keep trying to send a log directory notification before shutting down.

https://cwiki.apache.org/confluence/display/KAFKA/KIP-858%3A+Handle+JBOD+broker+disk+failure+in+KRaft

Best,

--
Igor

Re: Apache Kafka 3.6.0 release

[Satish Duggana]

Thanks for the update, Mayank Shekhar.

On Tue, 25 Jul 2023 at 15:48, Mayank Shekhar Narula
 wrote:
>
> Hi Satish
>
> Heads up KIP-951 is under voting. This could be delayed by a day or two for
> the 3.6 KIP deadline.
>
> On Tue, Jul 25, 2023 at 4:19 AM Satish Duggana 
> wrote:
>
> > Thanks Colin for the update on the mentioned KIPs.
> >
> > ~Satish.
> >
> > On Mon, 24 Jul 2023 at 23:09, Colin McCabe  wrote:
> > >
> > > Hi Satish,
> > >
> > > I removed "KIP-866 ZooKeeper to KRaft Migration" from the list of
> > pending KIPs, since that one was shipped in 3.4. I added "KIP-868 Metadata
> > Transactions", since we are planning on implementing this in 3.6. (The KIP
> > was approved a while ago, but not yet shipped.)
> > >
> > > I also added "KIP-938: Add more metrics for measuring KRaft
> > performance," which is a new KIP we are implemeting in 3.6 (The JIRA is
> > open now for review.) Same for KIP-919 which is being voted on now.
> > >
> > > best,
> > > Colin
> > >
> > > On Mon, Jul 24, 2023, at 03:59, Satish Duggana wrote:
> > > > A gentle reminder on the KIP freeze date: 26th Jul. Please try to
> > > > close discussion/vote threads asap.
> > > >
> > > > Thanks,
> > > > Satish.
> > > >
> > > > On Sun, 23 Jul 2023 at 11:10, Satish Duggana 
> > wrote:
> > > >>
> > > >> Thanks Colov/Divij for adding the KIP-952. I do not think it is a
> > > >> blocker for 3.6.0. We can discuss the KIP in the respective thread.
> > > >>
> > > >> ~Satish.
> > > >>
> > > >> On Sun, 23 Jul 2023 at 07:21, Satish Duggana <
> > satish.dugg...@gmail.com> wrote:
> > > >> >
> > > >> > Thanks ShunKang for the update. I added both the KIPs to the wiki.
> > > >> > Please feel free to update the wiki with the latest.
> > > >> >
> > > >> > ~Satish.
> > > >> >
> > > >> > On Sat, 22 Jul 2023 at 22:50, ShunKang Lin <
> > linshunkang@gmail.com> wrote:
> > > >> > >
> > > >> > > Hi Satish,
> > > >> > >
> > > >> > > Could we add "KIP-863: Reduce CompletedFetch#parseRecord() memory
> > copy" [1]
> > > >> > > and "KIP-872: Add Serializer#serializeToByteBuffer() to reduce
> > memory
> > > >> > > copying" [2] to the release plan?
> > > >> > > Thanks!
> > > >> > >
> > > >> > > [1]
> > > >> > >
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=225152035
> > > >> > >
> > > >> > > [2]
> > > >> > >
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=228495828
> > > >> > > I would appreciate a few more reviews on the pull request (
> > > >> > > https://github.com/apache/kafka/pull/12685) for KIP-872.
> > > >> > >
> > > >> > > Best,
> > > >> > > ShunKang
> > > >> > >
> > > >> > > Divij Vaidya  于2023年7月22日周六 20:06写道：
> > > >> > >
> > > >> > > > Hi Satish
> > > >> > > >
> > > >> > > > I have added the following accepted KIPs to the release plan.
> > Please let me
> > > >> > > > know if something requires a change.
> > > >> > > >
> > > >> > > > Accepted KIPs -
> > > >> > > >
> > > >> > > > 1.
> > > >> > > >
> > > >> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-852%3A+Optimize+calculation+of+size+for+log+in+remote+tier
> > > >> > > >
> > > >> > > > 2.
> > > >> > > >
> > > >> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-937%3A+Improve+Message+Timestamp+Validation
> > > >> > > >
> > > >> > > >
> > > >> > > > Pending discussion KIP which I believe is important to be
> > merged into 3.6 -
> > > >> > > >
> > > >> > > > 3.
> > > >> > > >
> > > >> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-952%3A+Regenerate+segment-aligned+producer+snapshots+when+upgrading+to+a+Kafka+version+supporting+Tiered+Storage
> > > >> > > >
> > > >> > > >
> > > >> > > > --
> > > >> > > > Divij Vaidya
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > > On Sat, Jul 22, 2023 at 6:41 AM Satish Duggana <
> > satish.dugg...@gmail.com>
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > > > Thanks Hao for the update on KIP-925.
> > > >> > > > >
> > > >> > > > > On Thu, 20 Jul 2023 at 23:05, Hao Li 
> > > >> > > > > 
> > wrote:
> > > >> > > > > >
> > > >> > > > > > Hi Satish,
> > > >> > > > > >
> > > >> > > > > > KIP-925 was accepted and currently under implementation. I
> > just added
> > > >> > > > it
> > > >> > > > > to
> > > >> > > > > > the release plan.
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-925%3A+Rack+aware+task+assignment+in+Kafka+Streams
> > > >> > > > > >
> > > >> > > > > > Thanks,
> > > >> > > > > > Hao
> > > >> > > > > >
> > > >> > > > > > On Thu, Jul 20, 2023 at 6:18 AM Christo Lolov <
> > christolo...@gmail.com>
> > > >> > > > > > wrote:
> > > >> > > > > >
> > > >> > > > > > > Hello!
> > > >> > > > > > >
> > > >> > > > > > > A couple of days ago I opened a new KIP for discussion -
> > KIP-952
> > > >> > > > [1]. I
> > > >> > > > > > > believe it might be a blocker for the release of 3.6.0,
> > but I wanted
> > > >> > > > to
> > > >> > > > > > > bring it up here for a decision on its urgency with 

Re: [VOTE] KIP-930: Tiered Storage Metrics

[Satish Duggana]

+1 for the KIP.

Thanks,
Satish.

On Tue, 25 Jul 2023 at 18:31, Kamal Chandraprakash
 wrote:
>
> +1 (non-binding)
>
> --
> Kamal
>
> On Tue, Jul 25, 2023 at 11:30 AM Abhijeet Kumar 
> wrote:
>
> > Hi All,
> >
> > I would like to start the vote for KIP-930 Tiered Storage Metrics.
> >
> > The KIP is here:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-930%3A+Tiered+Storage+Metrics
> >
> > Regards
> > Abhijeet.
> >

Re: Re: [DISCUSS] KIP-951: Leader discovery optimisations for the client

[Ismael Juma]

Thanks Crispin!

Ismael

On Mon, Jul 24, 2023 at 8:16 PM Crispin Bernier
 wrote:

> I updated the wiki to include both results along with their average.
>
> Thank you,
> Crispin
>
> On Mon, Jul 24, 2023 at 10:58 AM Ismael Juma  wrote:
>
> > Hi Crispin,
> >
> > One additional question, the wiki says "The results are averaged over 2
> > runs.". Can you please provide some measure of variance in the
> > distribution, i.e. were both results similar to each other for both
> cases?
> >
> > Ismael
> >
> > On Fri, Jul 21, 2023 at 11:31 AM Ismael Juma  wrote:
> >
> > > Thanks for the update Crispin - very helpful to have actual performance
> > > data. 2-5% for the default configuration is a bit on the low side for
> > this
> > > kind of proposal.
> > >
> > > Ismael
> > >
> > > On Thu, Jul 20, 2023 at 11:33 PM Crispin Bernier
> > >  wrote:
> > >
> > >> Benchmark numbers have been posted on the KIP, please review.
> > >>
> > >> On 2023/07/20 13:03:00 Mayank Shekhar Narula wrote:
> > >> > Jun
> > >> >
> > >> > Thanks for the feedback.
> > >> >
> > >> > Numbers to follow.
> > >> >
> > >> > If we don't plan to
> > >> > > bump up the FetchResponse version, we could just remove the
> > reference
> > >> to
> > >> > > version 16.
> > >> >
> > >> > Fixed.
> > >> >
> > >> > On Thu, Jul 20, 2023 at 1:28 AM Jun Rao  >
> > >> wrote:
> > >> >
> > >> > > Hi, Mayank,
> > >> > >
> > >> > > Thanks for the KIP. I agree with others that it would be useful to
> > >> see the
> > >> > > performance results. Otherwise, just a minor comment. If we don't
> > >> plan to
> > >> > > bump up the FetchResponse version, we could just remove the
> > reference
> > >> to
> > >> > > version 16.
> > >> > >
> > >> > > Jun
> > >> > >
> > >> > > On Wed, Jul 19, 2023 at 2:31 PM Mayank Shekhar Narula <
> > >> > > mayanks.nar...@gmail.com> wrote:
> > >> > >
> > >> > > > Luke
> > >> > > >
> > >> > > > Thanks for the interest in the KIP.
> > >> > > >
> > >> > > > But what if the consumer was fetching from the follower?
> > >> > > >
> > >> > > > We already include `PreferredReadReplica` in the fetch response.
> > >> > > > > Should we put the node info of PreferredReadReplica under this
> > >> case,
> > >> > > > > instead of the leader's info?
> > >> > > > >
> > >> > > >
> > >> > > > PreferredReadReplica is the decided on the leader. Looking at
> the
> > >> Java
> > >> > > > client code, AbstractFetch::selectReadReplica, first fetch
> request
> > >> goes
> > >> > > to
> > >> > > > Leader of the partition -> Sends back PreferredReadReplica ->
> Next
> > >> fetch
> > >> > > > uses PreferredReadReplica. So as long as leader is available,
> > >> > > > PreferredReadReplica would be found in subsequent fetches.
> > >> > > >
> > >> > > > Also, under this case, should we include the leader's info in
> the
> > >> > > response?
> > >> > > >
> > >> > > >
> > >> > > > In this case, I think the follower would fail the fetch if it
> > knows
> > >> a
> > >> > > > different leader. If the follower knows a newer leader, it would
> > >> return
> > >> > > new
> > >> > > > leader information in the response, for the client to act on.
> > >> > > >
> > >> > > >
> > >> > > > Will we include the leader/node info in the response when having
> > >> > > > > `UNKNOWN_LEADER_EPOCH` error?
> > >> > > >
> > >> > > >
> > >> > > > My understanding is UNKNOWN_LEADER_EPOCH when a request from a
> > >> client
> > >> > > has a
> > >> > > > newer epoch than the broker. So the client is already up to date
> > on
> > >> new
> > >> > > > leader information, it's the broker that has the catching up to
> > do.
> > >> I
> > >> > > think
> > >> > > > there might be some optimisations to make sure the broker
> > refreshes
> > >> its
> > >> > > > metadata quickly, so it can quickly recover to handle requests
> > that
> > >> > > > previously returned UNKNOWN_LEADER_EPOCH. But this work is
> outside
> > >> the
> > >> > > > scope of this KIP, as for now this KIP focusses on client-side
> > >> > > > optimisations.
> > >> > > >
> > >> > > > Mayank
> > >> > > >
> > >> > > > On Tue, Jul 18, 2023 at 8:51 AM Luke Chen 
> > wrote:
> > >> > > >
> > >> > > > > Hi Mayank,
> > >> > > > >
> > >> > > > > Thanks for the KIP!
> > >> > > > >
> > >> > > > > Some questions:
> > >> > > > > 1. I can see most of the cases we only care about consumer
> fetch
> > >> from
> > >> > > the
> > >> > > > > leader.
> > >> > > > > But what if the consumer was fetching from the follower?
> > >> > > > > We already include `PreferredReadReplica` in the fetch
> response.
> > >> > > > > Should we put the node info of PreferredReadReplica under this
> > >> case,
> > >> > > > > instead of the leader's info?
> > >> > > > > Also, under this case, should we include the leader's info in
> > the
> > >> > > > response?
> > >> > > > >
> > >> > > > > 2. Will we include the leader/node info in the response when
> > >> having
> > >> > > > > `UNKNOWN_LEADER_EPOCH` error?
> > >> > > > > I think it's fine we ignore the `UNKNOWN_LEADER_EPOCH` error
> > >> since 

Re: [DISCUSS] KIP-954: expand default DSL store configuration to custom types

[Almog Gavra]

Glad you like my KIP-secretary skills ;)

A2. I'm definitely happy to take your suggestion here and not do anything
special w.r.t. Versioned stores, I think it makes sense especially if we
consider them implementation details of a specific store type.

At EOD I'll update the KIP with all of these changes and if the
discussion is silent I'll open a vote tomorrow morning.

Cheers,
Almog

On Mon, Jul 24, 2023 at 2:02 PM Sophie Blee-Goldman 
wrote:

> Awesome summary (seriously) -- would you kindly offer your organizational
> skills to every ongoing KIP from henceforth? We need you :P
>
> A few answers/comments:
>
> A2: I think there is a 3rd sub-option here, which is to leave
> versioned-ness out of this KIP entirely, return only the non-versioned
> stores for now, and then switch over to the versioned stores (only) when
> the time comes to flip the switch on making them the default across the
> DSL. This has the advantage of retaining the current behavior/semantics and
> provides a clear way to transition smoothly in the future, since it seems
> we will want to cut to all versioned state stores rather than offer users a
> choice between versioned or non-versioned stores going forward (similar to
> how we only offer timestamped stores presently, and have completely
> replaced non-timestamped stores in the DSL.) . In both the timestamped and
> versioned cases, the old stores are/will still be available or accessible
> to users via the bare StoreSuppliers, should they somehow desire or require
> the old store type. Ultimately, I think either this or option (1) would be
> preferable, though I think it should be up to Matthias or anyone else
> involved in the versioned stores feature to decide which approach makes
> sense in the context of that feature's future plans.
>
> A3: sounds reasonable to me
>
> A5: Also sounds fine to me, though I'll let others chime in with/if they
> have an alternative suggestion/preference. I guess the other contender
> would be something like DSLStoreImpl or something like that?
>
>
>
> On Mon, Jul 24, 2023 at 9:36 AM Almog Gavra  wrote:
>
> > Lots of thoughts! Happy to see the thriving discussion on this post -
> lots
> > going on so I'm trying to enumerate them to keep things organized (prefix
> > "A" for "Almog" so we can use numbers in responses for other things ;P).
> >
> > A1. Question around closing implementation gaps (e.g. no rocks based
> > suppression store)
> > A2. Specifically how to handle Versioned stores
> > A3. Configuration (new config/reuse old one + new one and ordering of
> > config resolution)
> > A4. Drawing a line between what is implementation detail (not exposed in
> > API) and what is customizable (exposed in API)
> > A5. Naming of StoreTypeSpec
> > A6. Param classes in StoreBuilders
> >
> > --
> >
> > Here are summaries for where it seems each of these stands (trying not to
> > add any additional opinion yet):
> >
> > A1. Sophie/Guozhang/Me (if I count hah!) seem to agree that it is worth
> > pushing this KIP through independently of the implementation gaps as it
> > doesn't seem to move the intermediate state further from the end state.
> > Matthias originally had some concerns.
> >
> > A2. There's questions around whether versioned stores should be their own
> > configurable option or whether they are an implementation detail that the
> > StoreSpec should decide. It seems like the discussion is converging here,
> > this should be an implementation detail.
> >
> > A3. Matthias/Guozhang prefer adding CUSTOM and then having an additional
> > config to determine the implementation. Sophie prefers deprecating the
> old
> > config. Guozhang additional suggested flipping the resolution order such
> > that the old config is only respected in a DefaultStoreSpec
> implementation.
> >
> > A4. This KIP (or rather, the discussion on the KIP) blurs the lines
> between
> > top level store types (KV, windowed, session) and the underlying
> > implementation of them (timestamped, versioned, kv-list). It seems
> everyone
> > is in alignment to ensure that we keep these two things separate and that
> > the line is clearly delineated in the text of the KIP.
> >
> > A5. Guozhang and Sophie agree that the current name StoreTypeSpec is
> > misleading, as it's really an implementation spec, not a type
> > specification.
> >
> > A6. Agreement that this is an improvement, Sophie believes this can be
> done
> > in a follow up but we should ensure our naming is good here so there's no
> > conflicts down the line.
> >
> > -
> >
> > Ok, phew! Hopefully that covers it all! Now for my thoughts, hopefully
> > wrapping up some of these discussions:
> >
> > A1.  @Matthias - are you still hesitant here? What would you need to be
> > convinced here?
> >
> > A2. Since we are all in agreement that versioned stores should be an
> > implementation detail, we have two options:
> >
> > (1) we can extend the KVParams to include a 

Re: [DISCUSS] KIP-959 Add BooleanConverter to Kafka Connect

[Chris Egerton]

Thanks Hector! LGTM

On Tue, Jul 25, 2023 at 11:47 AM Hector Geraldino (BLOOMBERG/ 919 3RD A) <
hgerald...@bloomberg.net> wrote:

> Thanks Chris for your quick reply.
>
> Your suggestions make sense, I amended the KIP and added a note to the
> class JavaDocs. Also added unit tests to the companion PR [
> https://github.com/apache/kafka/pull/14093], and will mark it as "Ready
> for Review" in a few.
>
> Cheers
>
> From: dev@kafka.apache.org At: 07/25/23 10:42:58 UTC-4:00To:
> dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-959 Add BooleanConverter to Kafka Connect
>
> Hi Hector,
>
> Thanks for the KIP! Really appreciate the tight scope, hoping this will be
> easy to review :)
>
> I only have one question: it looks like our existing primitive converters
> (string converter + subclasses of NumberConverter) are hardcoded to play
> nicely with null values during deserialization by always providing an
> optional schema. If that's the intent with this KIP, can we specify that
> explicitly? (Could be as simple as saying "the schema returned during
> deserialization will always be an optional boolean schema" with a link to
>
> https://kafka.apache.org/35/javadoc/org/apache/kafka/connect/data/Schema.html#OP
> TIONAL_BOOLEAN_SCHEMA).
> I don't think we have to say anything else about null handling since
> FWICT the rest is already handled by the BooleanSerializer and
> BooleanDeserializer introduced in KIP-907.
>
> Cheers,
>
> Chris
>
> On Tue, Jul 25, 2023 at 9:52 AM Hector Geraldino (BLOOMBERG/ 919 3RD A) <
> hgerald...@bloomberg.net> wrote:
>
> > Hi everyone,
> >
> > I'd like to start a discussion of KIP-959, which aims to add a
> > BooleanConverter to Kafka Connect:
> >
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-959%3A+Add+BooleanConverte
> r+to+Kafka+Connect
> >
> > This KIP is a counterpart of KIP-907: Add Boolean Serde to public
> > interface [
> >
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-907%3A+Add+Boolean+Serde+t
> o+public+interface],
> > which added Boolean SerDes to the Kafka serialization APIs.
> >
> > The scope of this KIP is very limited, and will help us close a small gap
> > that exists on the list of included converters for connect's "primitive"
> > types.
> >
> > Looking forward for your feedback.
> >
> > Regards,
> > Hector
>
>
>

Re: [DISCUSS] KIP-959 Add BooleanConverter to Kafka Connect

[Hector Geraldino (BLOOMBERG/ 919 3RD A)]

Thanks Chris for your quick reply.

Your suggestions make sense, I amended the KIP and added a note to the class 
JavaDocs. Also added unit tests to the companion PR 
[https://github.com/apache/kafka/pull/14093], and will mark it as "Ready for 
Review" in a few.

Cheers

From: dev@kafka.apache.org At: 07/25/23 10:42:58 UTC-4:00To:  
dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-959 Add BooleanConverter to Kafka Connect

Hi Hector,

Thanks for the KIP! Really appreciate the tight scope, hoping this will be
easy to review :)

I only have one question: it looks like our existing primitive converters
(string converter + subclasses of NumberConverter) are hardcoded to play
nicely with null values during deserialization by always providing an
optional schema. If that's the intent with this KIP, can we specify that
explicitly? (Could be as simple as saying "the schema returned during
deserialization will always be an optional boolean schema" with a link to
https://kafka.apache.org/35/javadoc/org/apache/kafka/connect/data/Schema.html#OP
TIONAL_BOOLEAN_SCHEMA).
I don't think we have to say anything else about null handling since
FWICT the rest is already handled by the BooleanSerializer and
BooleanDeserializer introduced in KIP-907.

Cheers,

Chris

On Tue, Jul 25, 2023 at 9:52 AM Hector Geraldino (BLOOMBERG/ 919 3RD A) <
hgerald...@bloomberg.net> wrote:

> Hi everyone,
>
> I'd like to start a discussion of KIP-959, which aims to add a
> BooleanConverter to Kafka Connect:
> 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-959%3A+Add+BooleanConverte
r+to+Kafka+Connect
>
> This KIP is a counterpart of KIP-907: Add Boolean Serde to public
> interface [
> 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-907%3A+Add+Boolean+Serde+t
o+public+interface],
> which added Boolean SerDes to the Kafka serialization APIs.
>
> The scope of this KIP is very limited, and will help us close a small gap
> that exists on the list of included converters for connect's "primitive"
> types.
>
> Looking forward for your feedback.
>
> Regards,
> Hector

Re: [DISCUSS] KIP-935: Extend AlterConfigPolicy with existing configurations

[Jorge Esteban Quilcate Otoya]

KIP is updated now:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-935%3A+Extend+AlterConfigPolicy+with+existing+configurations

Looking forward to your feedback,

Many thanks,
Jorge.

On Tue, 25 Jul 2023 at 16:59, Jorge Esteban Quilcate Otoya <
quilcate.jo...@gmail.com> wrote:

> Hi Colin, sorry for the belated follow up.
>
> If I understand correctly, on your latest reply proposed to have a new
> API. From the proposed alternatives, I lean towards the first alternative
> proposed with 2 config maps, old (before-alter) and new (after-alter).
> Deleting a config is effectively returning to the default value, then users
> can use the old value and compare against default if new is null.
>
> This would require a bit broader changes, starting with a new config. I
> will work on the KIP updates considering: `AlterConfigV2Policy` interface,
> and config `alter.config.policy.v2.class.name`. Let me know if there's
> any issues with this; otherwise I will update the mail thread once the KIP
> is updated.
>
> Many thanks,
> Jorge.
>
> On Tue, 20 Jun 2023 at 11:56, Jorge Esteban Quilcate Otoya <
> quilcate.jo...@gmail.com> wrote:
>
>> Thanks Colin! You're right. I started this KIP only thinking on the
>> latest incremental API, and haven't thought much on the legacy one.
>>
>> After taking a another look at both APIs, I can see some inconsistencies
>> on how the policies are applied in each case. I have added a section
>> "Current workflow" [1] to the current proposal to summarize how alter
>> config works in both cases (legacy and incremental) and for both back-ends
>> (ZK, KRaft).
>>
>> In summary:
>> - On Legacy Alter Config, the set of changes proposed is the same as the
>> new config with the difference that null values are removed from the new
>> config.
>> - On Incremental Alter Config, the set of changes proposed is not the
>> same as the new config. It only contains explicit changes to the config
>> - Implicit deletes are a set of configs inferred on legacy alter config
>> when no value is provided but it exists on the current config
>> - Even though alter config policy receives the "requested"
>> configurations, these have 2 different meanings depending on the API used
>> to update configs.
>>   - When validating policies on Legacy Alter Config, it means: requested
>> changes that is equal to new config state including explicit deletes
>>   - When validating policies on Incremental Alter Config, it means: only
>> requested changes including explicit deletes but without any other config
>> from current or new status
>>   - Plugin implementations *do not know which one are they actually
>> dealing with*, and as incremental (new) API becomes broadly adopted, then
>> current status configs not included in the request are not considered.
>>
>> The problem is a bit larger than the one framed on the motivation. It's
>> not only that we don't have the current configs to compare with; but
>> depending on the API used to alter configs we may have them or not.
>>
>> Is this assessment correct?
>> If it is, then we may discuss approaching this issue as a bug instead. We
>> could consider aligning the semantics of the configs passed to the policy.
>> At the moment the "requested configs" are passed to policies when either
>> API is called, but both have _different meaning depending on which API is
>> used_. We could instead align the meaning, and pass the "new configs,
>> including explicit deletes" as we do on legacy when doing incremental
>> updates as well.
>>
>> Looking forward to your feedback and many thanks again!
>> Jorge.
>>
>> [1]
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-935%3A+Extend+AlterConfigPolicy+with+existing+configurations#KIP935:ExtendAlterConfigPolicywithexistingconfigurations-Currentworkflow
>>
>> On Thu, 15 Jun 2023 at 22:07, Colin McCabe  wrote:
>>
>>> Hi Jorge,
>>>
>>> I appreciate you trying to solve the issue. However, from the
>>> perspective of someone using the plugin API, it's quite messy: what is the
>>> difference between "proposed" and "resulting"? They sound the same.
>>>
>>> I think there are two APIs that make sense:
>>>
>>> 1. A (prev, next) based one where you just get the previous set of
>>> configs, and the new one, and can draw your own conclusions
>>>
>>> 2. A (prev, changed, removed) one where you get the previous set of
>>> configs, plus the changes (additions or modifications), and deletions.
>>>
>>> 3. Same as 2 but you have a "changed" map whose values are Optionals,
>>> and express deletions as Optional.empty
>>>
>>> The old API should just stay the same, bugs and all, for compatibility
>>> reasons. But for the new API we should choose one of the above, I think.
>>> I'm not completely sure which...
>>>
>>> best,
>>> Colin
>>>
>>> On Mon, Jun 12, 2023, at 07:08, Jorge Esteban Quilcate Otoya wrote:
>>> > Thanks Colin! You're right. I have added some notes about this to the
>>> KIP,
>>> > and clarify how this KIP is related to legacy and incremental 

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2035

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 296972 lines...]
Gradle Test Run :streams:integrationTest > Gradle Test Executor 179 > 
RestoreIntegrationTest > shouldSuccessfullyStartWhenLoggingDisabled(boolean) > 
[2] false PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 179 > 
RestoreIntegrationTest > shouldRestoreStateFromChangelogTopic(boolean) > [1] 
true STARTED
https://ge.apache.org/s/rkbcharmj4pig


See the profiling report at: 
file:///home/jenkins/workspace/Kafka_kafka_trunk/build/reports/profile/profile-2023-07-25-12-25-05.html
A fine-grained performance profile is available: use the --scan option.
[Pipeline] junit
Recording test results

Gradle Test Run :streams:integrationTest > Gradle Test Executor 179 > 
RestoreIntegrationTest > shouldRestoreStateFromChangelogTopic(boolean) > [1] 
true PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 179 > 
RestoreIntegrationTest > shouldRestoreStateFromChangelogTopic(boolean) > [2] 
false STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 179 > 
RestoreIntegrationTest > shouldRestoreStateFromChangelogTopic(boolean) > [2] 
false PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 179 > 
SmokeTestDriverIntegrationTest > shouldWorkWithRebalance(boolean) > [1] true 
STARTED
[Checks API] No suitable checks publisher found.
[Pipeline] echo
Skipping Kafka Streams archetype test for Java 11
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // stage
[Pipeline] }

Gradle Test Run :streams:integrationTest > Gradle Test Executor 178 > 
RestoreIntegrationTest > shouldRestoreNullRecord() PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 178 > 
RestoreIntegrationTest > shouldRestoreStateFromSourceTopic(boolean) > [1] true 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 178 > 
RestoreIntegrationTest > shouldRestoreStateFromSourceTopic(boolean) > [1] true 
PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 178 > 
RestoreIntegrationTest > shouldRestoreStateFromSourceTopic(boolean) > [2] false 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 178 > 
RestoreIntegrationTest > shouldRestoreStateFromSourceTopic(boolean) > [2] false 
PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 178 > 
RestoreIntegrationTest > shouldSuccessfullyStartWhenLoggingDisabled(boolean) > 
[1] true STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 178 > 
RestoreIntegrationTest > shouldSuccessfullyStartWhenLoggingDisabled(boolean) > 
[1] true PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 178 > 
RestoreIntegrationTest > shouldSuccessfullyStartWhenLoggingDisabled(boolean) > 
[2] false STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 178 > 
RestoreIntegrationTest > shouldSuccessfullyStartWhenLoggingDisabled(boolean) > 
[2] false PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 178 > 
RestoreIntegrationTest > shouldRestoreStateFromChangelogTopic(boolean) > [1] 
true STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 178 > 
RestoreIntegrationTest > shouldRestoreStateFromChangelogTopic(boolean) > [1] 
true PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 178 > 
RestoreIntegrationTest > shouldRestoreStateFromChangelogTopic(boolean) > [2] 
false STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 178 > 
RestoreIntegrationTest > shouldRestoreStateFromChangelogTopic(boolean) > [2] 
false PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 178 > 
SmokeTestDriverIntegrationTest > shouldWorkWithRebalance(boolean) > [1] true 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 179 > 
SmokeTestDriverIntegrationTest > shouldWorkWithRebalance(boolean) > [1] true 
PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 179 > 
SmokeTestDriverIntegrationTest > shouldWorkWithRebalance(boolean) > [2] false 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 178 > 
SmokeTestDriverIntegrationTest > shouldWorkWithRebalance(boolean) > [1] true 
PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 178 > 
SmokeTestDriverIntegrationTest > shouldWorkWithRebalance(boolean) > [2] false 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 179 > 
SmokeTestDriverIntegrationTest > shouldWorkWithRebalance(boolean) > [2] false 
PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 179 > 
StoreQueryIntegrationTest > shouldQuerySpecificActivePartitionStores() 

Re: [DISCUSS] KIP-959 Add BooleanConverter to Kafka Connect

[Chris Egerton]

Hi Hector,

Thanks for the KIP! Really appreciate the tight scope, hoping this will be
easy to review :)

I only have one question: it looks like our existing primitive converters
(string converter + subclasses of NumberConverter) are hardcoded to play
nicely with null values during deserialization by always providing an
optional schema. If that's the intent with this KIP, can we specify that
explicitly? (Could be as simple as saying "the schema returned during
deserialization will always be an optional boolean schema" with a link to
https://kafka.apache.org/35/javadoc/org/apache/kafka/connect/data/Schema.html#OPTIONAL_BOOLEAN_SCHEMA).
I don't think we have to say anything else about null handling since
FWICT the rest is already handled by the BooleanSerializer and
BooleanDeserializer introduced in KIP-907.

Cheers,

Chris

On Tue, Jul 25, 2023 at 9:52 AM Hector Geraldino (BLOOMBERG/ 919 3RD A) <
hgerald...@bloomberg.net> wrote:

> Hi everyone,
>
> I'd like to start a discussion of KIP-959, which aims to add a
> BooleanConverter to Kafka Connect:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-959%3A+Add+BooleanConverter+to+Kafka+Connect
>
> This KIP is a counterpart of KIP-907: Add Boolean Serde to public
> interface [
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-907%3A+Add+Boolean+Serde+to+public+interface],
> which added Boolean SerDes to the Kafka serialization APIs.
>
> The scope of this KIP is very limited, and will help us close a small gap
> that exists on the list of included converters for connect's "primitive"
> types.
>
> Looking forward for your feedback.
>
> Regards,
> Hector

[jira] [Created] (KAFKA-15249) Verify Connect test-plugins artifact is published to Maven Central

[Chris Egerton (Jira)]

Chris Egerton created KAFKA-15249:
-

 Summary: Verify Connect test-plugins artifact is published to 
Maven Central
 Key: KAFKA-15249
 URL: https://issues.apache.org/jira/browse/KAFKA-15249
 Project: Kafka
  Issue Type: Task
Affects Versions: 3.6.0
Reporter: Chris Egerton
Assignee: Chris Egerton
 Fix For: 3.6.0


In KAFKA-14759 we created a separate {{connect/test-plugins}} module to store 
all testing-only Connect plugins and removed those plugins from existing 
Connect modules.

These testing-only plugins are intentionally excluded from the project's 
release file (which can be generated with {{{}./gradlew releaseTarGz{}}}) 
however, some users may still be relying on them for testing environments.

Although we should refrain from distributing these testing-only plugins with 
our out-of-the-box distribution of Connect, we should still ensure that they're 
available on an opt-in basis to users who would like to continue using them. 
This can be accomplished by publishing them to [Maven 
Central|https://search.maven.org/], like we do with our other modules.

This will probably happen automatically during the next release (3.6.0) with no 
further action required. This ticket is just here as a reminder to verify that 
the artifacts are present in the staging Maven repo when release candidates are 
published for voting.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-935: Extend AlterConfigPolicy with existing configurations

[Jorge Esteban Quilcate Otoya]

Hi Colin, sorry for the belated follow up.

If I understand correctly, on your latest reply proposed to have a new API.
>From the proposed alternatives, I lean towards the first alternative
proposed with 2 config maps, old (before-alter) and new (after-alter).
Deleting a config is effectively returning to the default value, then users
can use the old value and compare against default if new is null.

This would require a bit broader changes, starting with a new config. I
will work on the KIP updates considering: `AlterConfigV2Policy` interface,
and config `alter.config.policy.v2.class.name`. Let me know if there's any
issues with this; otherwise I will update the mail thread once the KIP is
updated.

Many thanks,
Jorge.

On Tue, 20 Jun 2023 at 11:56, Jorge Esteban Quilcate Otoya <
quilcate.jo...@gmail.com> wrote:

> Thanks Colin! You're right. I started this KIP only thinking on the latest
> incremental API, and haven't thought much on the legacy one.
>
> After taking a another look at both APIs, I can see some inconsistencies
> on how the policies are applied in each case. I have added a section
> "Current workflow" [1] to the current proposal to summarize how alter
> config works in both cases (legacy and incremental) and for both back-ends
> (ZK, KRaft).
>
> In summary:
> - On Legacy Alter Config, the set of changes proposed is the same as the
> new config with the difference that null values are removed from the new
> config.
> - On Incremental Alter Config, the set of changes proposed is not the same
> as the new config. It only contains explicit changes to the config
> - Implicit deletes are a set of configs inferred on legacy alter config
> when no value is provided but it exists on the current config
> - Even though alter config policy receives the "requested" configurations,
> these have 2 different meanings depending on the API used to update configs.
>   - When validating policies on Legacy Alter Config, it means: requested
> changes that is equal to new config state including explicit deletes
>   - When validating policies on Incremental Alter Config, it means: only
> requested changes including explicit deletes but without any other config
> from current or new status
>   - Plugin implementations *do not know which one are they actually
> dealing with*, and as incremental (new) API becomes broadly adopted, then
> current status configs not included in the request are not considered.
>
> The problem is a bit larger than the one framed on the motivation. It's
> not only that we don't have the current configs to compare with; but
> depending on the API used to alter configs we may have them or not.
>
> Is this assessment correct?
> If it is, then we may discuss approaching this issue as a bug instead. We
> could consider aligning the semantics of the configs passed to the policy.
> At the moment the "requested configs" are passed to policies when either
> API is called, but both have _different meaning depending on which API is
> used_. We could instead align the meaning, and pass the "new configs,
> including explicit deletes" as we do on legacy when doing incremental
> updates as well.
>
> Looking forward to your feedback and many thanks again!
> Jorge.
>
> [1]
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-935%3A+Extend+AlterConfigPolicy+with+existing+configurations#KIP935:ExtendAlterConfigPolicywithexistingconfigurations-Currentworkflow
>
> On Thu, 15 Jun 2023 at 22:07, Colin McCabe  wrote:
>
>> Hi Jorge,
>>
>> I appreciate you trying to solve the issue. However, from the perspective
>> of someone using the plugin API, it's quite messy: what is the difference
>> between "proposed" and "resulting"? They sound the same.
>>
>> I think there are two APIs that make sense:
>>
>> 1. A (prev, next) based one where you just get the previous set of
>> configs, and the new one, and can draw your own conclusions
>>
>> 2. A (prev, changed, removed) one where you get the previous set of
>> configs, plus the changes (additions or modifications), and deletions.
>>
>> 3. Same as 2 but you have a "changed" map whose values are Optionals, and
>> express deletions as Optional.empty
>>
>> The old API should just stay the same, bugs and all, for compatibility
>> reasons. But for the new API we should choose one of the above, I think.
>> I'm not completely sure which...
>>
>> best,
>> Colin
>>
>> On Mon, Jun 12, 2023, at 07:08, Jorge Esteban Quilcate Otoya wrote:
>> > Thanks Colin! You're right. I have added some notes about this to the
>> KIP,
>> > and clarify how this KIP is related to legacy and incremental alter
>> config
>> > APIs.
>> >
>> > Let me know if there's any gaps on the current proposal.
>> >
>> > Many thanks,
>> > Jorge.
>> >
>> > On Mon, 12 Jun 2023 at 11:04, Colin McCabe  wrote:
>> >
>> >> See KAFKA-14195. Some deletions are not handled correctly. And this
>> cannot
>> >> be fixed without a kip because of backwards compatibility.
>> >>
>> >> Colin
>> >>
>> >> On Wed, Jun 7, 2023, at 

Re: [DISCUSS] KIP-953: partition method to be overloaded to accept headers as well.

[Jack Tomy]

Hey @Sagar

Thanks again for the review.
1. "a null headers value is equivalent to invoking the older partition
method", this is not true. If someone makes an implementation and the
headers come as null, still the new implementation will take effect.
Instead I have added : "Not overriding this method in the Partitioner
interface has the same behaviour as using the existing method."
2. Corrected.

Hey @Sagar and everyone,
Please have a look at the new version and share your thoughts.
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263424937
Like Sagar mentioned, I would also request more people who have more
context on clients to chime in.


On Tue, Jul 25, 2023 at 2:58 PM Sagar  wrote:

> Hi Jack,
>
> Thanks I have a couple of final comments and then I am good.
>
> 1) Can you elaborate on the Javadocs of the partition headers argument to
> specify that a null headers value is equivalent to invoking the older
> partition method? It is apparent but generally good to call out.
> 2) In the Compatibility section, you have mentioned backward comparable. I
> believe it should be *backward compatible change.*
>
> I don't have other comments. Post this, probably someone else who has more
> context on Clients can also chime in on this before we can move this to
> Voting.
>
> Thanks!
> Sagar.
>
>
> On Sat, Jul 22, 2023 at 10:09 AM Jack Tomy  wrote:
>
> > Hey @Sagar,
> >
> > Thank you again for the response and feedback.
> >
> >1. Though the ask wasn't very clear to me I have attached the Javadoc
> as
> >per your suggestion. Please have a look and let me know if this meets
> > the
> >expectations.
> >2. Done.
> >3. Done
> >4. Done
> >
> > Hey @Sagar and everyone,
> > Please have a look at the new version and share your thoughts.
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263424937
> >
> > On Thu, Jul 20, 2023 at 9:46 PM Sagar  wrote:
> >
> > > Thanks Jack for the updates.
> > >
> > > Some more feedback:
> > >
> > > 1) It would be better if you can add the Javadoc in the Public
> interfaces
> > > section. That is a general practice used which gives the readers of the
> > KIP
> > > a high level idea of the Public Interfaces.
> > >
> > > 2) In the proposed section, the bit about marking headers as read only
> > > seems like an implementation detail This can generally be avoided in
> > KIPs.
> > >
> > > 3) Also, in the Deprecation section, can you mention again that this
> is a
> > > backward compatible change and the reason for it (already done in the
> > > Proposed Changes section).
> > >
> > > 4) In the Testing Plan section, there is still the KIP template bit
> > copied
> > > over. That can be removed.
> > >
> > > Thanks!
> > > Sagar.
> > >
> > >
> > > On Thu, Jul 20, 2023 at 2:48 PM Jack Tomy 
> wrote:
> > >
> > > > Hey Everyone,
> > > >
> > > > Please consider this as a reminder and share your feedback. Thank
> you.
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263424937
> > > >
> > > > On Tue, Jul 18, 2023 at 5:43 PM Jack Tomy 
> > wrote:
> > > >
> > > > > Hey @Sagar,
> > > > >
> > > > > Thank you for the response and feedback.
> > > > >
> > > > >1. Done
> > > > >2. Yeah, that was a mistake from my end. Corrected.
> > > > >3. Can you please elaborate this, I have added the java doc
> along
> > > with
> > > > >the code changes. Should I paste the same in KIP too?
> > > > >4. Moved.
> > > > >5. I have added one more use case, it is actually helpful in any
> > > > >situation where you want to pass some information to partition
> > > method
> > > > but
> > > > >don't have to have it in the key or value.
> > > > >6. Added.
> > > > >
> > > > >
> > > > > Hey @Sagar and everyone,
> > > > > Please have a look at the new version and share your thoughts.
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263424937
> > > > >
> > > > >
> > > > > On Tue, Jul 18, 2023 at 9:53 AM Sagar 
> > > wrote:
> > > > >
> > > > >> Hi Jack,
> > > > >>
> > > > >> Thanks for the KIP! Seems like an interesting idea. I have some
> > > > feedback:
> > > > >>
> > > > >> 1) It would be great if you could clean up the text that seems to
> > > mimic
> > > > >> the
> > > > >> KIP template. It is generally not required in the KIP.
> > > > >>
> > > > >> 2) In the Public Interfaces where you mentioned *Partitioner
> method
> > in
> > > > >> **org/apache/kafka/clients/producer
> > > > >> will have the following update*, I believe you meant the
> Partitioner
> > > > >> *interface*?
> > > > >>
> > > > >> 3) Staying on Public Interface, it is generally preferable to add
> a
> > > > >> Javadocs section along with the newly added method. You could also
> > > > >> describe
> > > > >> the behaviour of it invoking the default existing method.
> > > > >>
> > > > >> 4) The option that is mentioned in the Rejected Alternatives,
> seems
> > > more
> > > > >> like a workaround to the 

[jira] [Created] (KAFKA-15248) Add BooleanConverter to Kafka Connect

[Hector Geraldino (Jira)]

Hector Geraldino created KAFKA-15248:


 Summary: Add BooleanConverter to Kafka Connect
 Key: KAFKA-15248
 URL: https://issues.apache.org/jira/browse/KAFKA-15248
 Project: Kafka
  Issue Type: Improvement
  Components: KafkaConnect
Reporter: Hector Geraldino
Assignee: Hector Geraldino


KIP-959: Add BooleanConverter to Kafka Connect -> 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-959%3A+Add+BooleanConverter+to+Kafka+Connect



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[DISCUSS] KIP-959 Add BooleanConverter to Kafka Connect

[Hector Geraldino (BLOOMBERG/ 919 3RD A)]

Hi everyone,

I'd like to start a discussion of KIP-959, which aims to add a BooleanConverter 
to Kafka Connect: 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-959%3A+Add+BooleanConverter+to+Kafka+Connect

This KIP is a counterpart of KIP-907: Add Boolean Serde to public interface 
[https://cwiki.apache.org/confluence/display/KAFKA/KIP-907%3A+Add+Boolean+Serde+to+public+interface],
 which added Boolean SerDes to the Kafka serialization APIs. 

The scope of this KIP is very limited, and will help us close a small gap that 
exists on the list of included converters for connect's "primitive" types.

Looking forward for your feedback.

Regards,
Hector

[jira] [Created] (KAFKA-15247) OutOfMemoryError in SaslClientAuthenticator during server restart

[Dave Crighton (Jira)]

Dave Crighton created KAFKA-15247:
-

 Summary: OutOfMemoryError in SaslClientAuthenticator during server 
restart 
 Key: KAFKA-15247
 URL: https://issues.apache.org/jira/browse/KAFKA-15247
 Project: Kafka
  Issue Type: Bug
  Components: clients
Affects Versions: 3.5.1
Reporter: Dave Crighton
 Attachments: 
0001-defensive-code-for-rogue-packets-during-server-resta.patch

We embed the Kafka client  in IBM App Connect Enterprise in order to provide 
Kafka consume and  produce functionality. This product is a little  bit like an 
app server in that it may host multiple workloads including some which may not 
use the Kafka functionality.

 

When the Kafka server is installed in an open shift environment we are seeing 
cases where the clients receive OutOfMemory errors due to single large (1.2Gb) 
byte buffers being allocated by the client.

 

>From research this appears to be a known issue when a plaintext client is 
>configured to attempt connection to a TLS secured endpoint however in this 
>instance we see successful communication  via TLS and then when the Kafka 
>server is restarted (or connectivity is broken) both the consumers and 
>producers can throw OutOfMemoryError's with the following stacks:

 

Producer



 

{{4XESTACKTRACE                at 
java/nio/HeapByteBuffer.(HeapByteBuffer.java:57(Compiled Code))}}
{{4XESTACKTRACE                at 
java/nio/ByteBuffer.allocate(ByteBuffer.java:335(Compiled Code))}}
{{4XESTACKTRACE                at 
org/apache/kafka/common/memory/MemoryPool$1.tryAllocate(MemoryPool.java:30(Compiled
 Code))}}
{{4XESTACKTRACE                at 
org/apache/kafka/common/network/NetworkReceive.readFrom(NetworkReceive.java:102(Compiled
 Code))}}
{{4XESTACKTRACE                at 
org/apache/kafka/common/security/authenticator/SaslClientAuthenticator.receiveResponseOrToken(SaslClientAuthenticator.java:475(Compiled
 Code))}}
{{4XESTACKTRACE                at 
org/apache/kafka/common/security/authenticator/SaslClientAuthenticator.receiveKafkaResponse(SaslClientAuthenticator.java:572(Compiled
 Code))}}
{{4XESTACKTRACE                at 
org/apache/kafka/common/security/authenticator/SaslClientAuthenticator.authenticate(SaslClientAuthenticator.java:250(Compiled
 Code))}}
{{4XESTACKTRACE                at 
org/apache/kafka/common/network/KafkaChannel.prepare(KafkaChannel.java:181(Compiled
 Code))}}
{{4XESTACKTRACE                at 
org/apache/kafka/common/network/Selector.pollSelectionKeys(Selector.java:543(Compiled
 Code))}}
{{4XESTACKTRACE                at 
org/apache/kafka/common/network/Selector.poll(Selector.java:481(Compiled 
Code))}}
{{4XESTACKTRACE                at 
org/apache/kafka/clients/NetworkClient.poll(NetworkClient.java:571(Compiled 
Code))}}
{{4XESTACKTRACE                at 
org/apache/kafka/clients/producer/internals/Sender.runOnce(Sender.java:328(Compiled
 Code))}}
{{4XESTACKTRACE                at 
org/apache/kafka/clients/producer/internals/Sender.run(Sender.java:243(Compiled 
Code))}}
{{4XESTACKTRACE                at java/lang/Thread.run(Thread.java:830)}}

 

Consumer

-

3XMTHREADINFO3   Java callstack:
4XESTACKTRACEat 
java/nio/HeapByteBuffer.(HeapByteBuffer.java:57)
4XESTACKTRACEat 
java/nio/ByteBuffer.allocate(ByteBuffer.java:335)
4XESTACKTRACEat 
org/apache/kafka/common/memory/MemoryPool$1.tryAllocate(MemoryPool.java:30)
4XESTACKTRACEat 
org/apache/kafka/common/network/NetworkReceive.readFrom(NetworkReceive.java:113)
4XESTACKTRACEat 
org/apache/kafka/common/security/authenticator/SaslClientAuthenticator.receiveResponseOrToken(SaslClientAuthenticator.java:475)
4XESTACKTRACEat 
org/apache/kafka/common/security/authenticator/SaslClientAuthenticator.receiveKafkaResponse(SaslClientAuthenticator.java:572)
4XESTACKTRACEat 
org/apache/kafka/common/security/authenticator/SaslClientAuthenticator.authenticate(SaslClientAuthenticator.java:250)
4XESTACKTRACEat 
org/apache/kafka/common/network/KafkaChannel.prepare(KafkaChannel.java:181)
4XESTACKTRACEat 
org/apache/kafka/common/network/Selector.pollSelectionKeys(Selector.java:543)
4XESTACKTRACEat 
org/apache/kafka/common/network/Selector.poll(Selector.java:481)
4XESTACKTRACEat 
org/apache/kafka/clients/NetworkClient.poll(NetworkClient.java:551)
4XESTACKTRACEat 
org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.poll(ConsumerNetworkClient.java:265)
4XESTACKTRACEat 
org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.poll(ConsumerNetworkClient.java:236)
4XESTACKTRACEat 
org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.poll(ConsumerNetworkClient.java:215)
4XESTACKTRACE

[VOTE] KIP-934: Add DeleteTopicPolicy

[Jorge Esteban Quilcate Otoya]

Hi All,

I'd like to start the vote for KIP-934: Add DeleteTopicPolicy:
https://cwiki.apache.org/confluence/x/-xE0Dw

Regards,
Jorge.

Re: [VOTE] KIP-930: Tiered Storage Metrics

[Kamal Chandraprakash]

+1 (non-binding)

--
Kamal

On Tue, Jul 25, 2023 at 11:30 AM Abhijeet Kumar 
wrote:

> Hi All,
>
> I would like to start the vote for KIP-930 Tiered Storage Metrics.
>
> The KIP is here:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-930%3A+Tiered+Storage+Metrics
>
> Regards
> Abhijeet.
>

Re: [VOTE] KIP-930: Tiered Storage Metrics

[Luke Chen]

+1 (binding) from me.

Thanks.
Luke

On Tue, Jul 25, 2023 at 7:51 PM Jorge Esteban Quilcate Otoya <
quilcate.jo...@gmail.com> wrote:

> +1 (non-binding)
>
> Thanks, Abhijeet!
>
>
> On Tue, 25 Jul 2023 at 14:22, Abhijeet Kumar 
> wrote:
>
> > Please find the updated link to the KIP:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-930%3A+Rename+ambiguous+Tiered+Storage+Metrics
> >
> > Updated the KIP as per our conversation on the discussion thread.
> >
> > On Tue, Jul 25, 2023 at 11:29 AM Abhijeet Kumar <
> > abhijeet.cse@gmail.com>
> > wrote:
> >
> > > Hi All,
> > >
> > > I would like to start the vote for KIP-930 Tiered Storage Metrics.
> > >
> > > The KIP is here:
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-930%3A+Tiered+Storage+Metrics
> > >
> > > Regards
> > > Abhijeet.
> > >
> > >
> >
> > --
> > Abhijeet.
> >
>

Re: KIP-919: Allow AdminClient to Talk Directly with the KRaft Controller Quorum

[Luke Chen]

Hi Colin,

Some more comments:
1. In the KIP, we mentioned "controller heartbeats", but it is not
explained anywhere.
I think "controller heartbeats" = controller registration", is that
correct?
If no, please explain more about it in the KIP.

2. Following this question:
> Which endpoint will the inactive controllers use to send the
ControllerRegistrationRequest?
> A: They will use the endpoint in controller.quorum.voters.
If the registration request will include controller.quorum.voters, why
bother sending this information to active controller again?
The active controller should already have all the controller.quorum.voters
when start up.
Any purpose of that design? For validation?

3. If a controller node is not part of `controller.quorum.voters`, when it
sends ControllerRegistrationRequest, what will we respond to it?

4. Nice and clear compatibility matrix!

Thank you.
Luke

On Sat, Jul 22, 2023 at 3:38 AM Colin McCabe  wrote:

> On Fri, Jul 21, 2023, at 09:43, José Armando García Sancio wrote:
> > Thanks for the KIP Colin. Apologies if some of these points have
> > already been made. I have not followed the discussion closely:
> >
> > 1. Re: Periodically, each controller will check that the controller
> > registration for its ID is as expected
> >
> > Does this need to be periodic? Can't the controller schedule this RPC,
> > retry etc, when it finds that the incarnation ID doesn't match its
> > own?
> >
>
> Hi José,
>
> Thanks for the reviews.
>
> David had the same question. I agree that it should be event-driven rather
> than periodic (except for retries, etc.)
>
> >
> > 2. Did you consider including the active controller's epoch in the
> > ControllerRegistrationRequest?
> >
> > This would allow the active controller to reject registration from
> > controllers that are not part of the active quorum and don't know the
> > latest controller epoch. This should mitigate some of the concerns you
> > raised in bullet point 1.
> >
>
> Good idea. I will add the active controller epoch to the registration
> request.
>
> >
> > 3. Which endpoint will the inactive controllers use to send the
> > ControllerRegistrationRequest?
> >
> > Will it use the first endpoint described in the cluster metadata
> > controller registration record? Or would it use the endpoint described
> > in the server configuration at controller.quorum.voters?
> >
>
> They will use the endpoint in controller.quorum.voters. In general, the
> endpoints from the registration are only used for responding to
> DESCRIBE_CLUSTER. Since, after all, we may not even have the registration
> endpoints when we start up.
>
> >
> > 4. Re: Raft integration in the rejected alternatives
> >
> > Yes, The KRaft layer needs to solve a similar problem like endpoint
> > discovery to support dynamic controller membership change. As you
> > point out the requirements are different and the set of information
> > that needs to be tracked is different. I think it is okay to use a
> > different solution for each of these problems.
>
> Yeah that was my feeling too. Thanks for taking a look.
>
> regards,
> Colin
>
> >
> > Thanks!
> > --
> > -José
>

Re: [VOTE] KIP-930: Tiered Storage Metrics

[Jorge Esteban Quilcate Otoya]

+1 (non-binding)

Thanks, Abhijeet!


On Tue, 25 Jul 2023 at 14:22, Abhijeet Kumar 
wrote:

> Please find the updated link to the KIP:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-930%3A+Rename+ambiguous+Tiered+Storage+Metrics
>
> Updated the KIP as per our conversation on the discussion thread.
>
> On Tue, Jul 25, 2023 at 11:29 AM Abhijeet Kumar <
> abhijeet.cse@gmail.com>
> wrote:
>
> > Hi All,
> >
> > I would like to start the vote for KIP-930 Tiered Storage Metrics.
> >
> > The KIP is here:
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-930%3A+Tiered+Storage+Metrics
> >
> > Regards
> > Abhijeet.
> >
> >
>
> --
> Abhijeet.
>

Re: [DISCUSS] KIP-910: Update Source offsets for Source Connectors without producing records

[Yash Mayya]

Hi Sagar,

Thanks for the updates. I had a few more follow up questions:

> I have added that a better way of doing that would be
> via KIP-875. Also, I didn't want to include any mechamisms
> for users to meddle with the offsets topic. Allowing tombstone
> records via this method would be akin to publishing tombstone
> records directly to the offsets topic which is not recommended
> generally.

KIP-875 would allow a way for cluster administrators and / or users to do
so manually externally whereas allowing tombstones in
SourceTask::updateOffsets would enable connectors to clean up offsets for
old / stale partitions without user intervention right? I'm not sure I
follow what you mean by "I didn't want to include any mechamisms for users
to meddle with the offsets topic" here? Furthermore, I'm not sure why
publishing tombstone records directly to the offsets topic would not be
recommended? Isn't that currently the only way to manually clean up offsets
for a source connector?

> It could be useful in a scenario where the offset of a partition
> doesn't update for some period of time. In such cases, the
> connector can do some kind of state tracking and update the
> offsets after the time period elapses.

I'm not sure I follow? In this case, won't the offsets argument passed
to SourceTask::updateOffsets *not *contain the source partition which
hasn't had an update for a long period of time? Wouldn't it make more sense
to reduce the surface of the API as Chris suggested and only allow adding
new partition offset pairs to the about to be committed offsets (since
there don't seem to be any use cases outlined for allowing connectors to
update offsets for source partitions that are already about to have an
offset be committed for)?

> All the records returned by the previous poll invocation
>  got processed successfully

Thanks for this clarification in the KIP, it looks like it does address the
offsets ordering issue. As to Chris' point about invoking
SourceTask::updateOffsets less frequently by calling it before offsets are
committed rather than in every poll loop iteration - I guess that would
make it a lot more tricky to address the ordering issue?


Thanks,
Yash

On Thu, Jul 20, 2023 at 9:50 PM Sagar  wrote:

> Hey All,
>
> Please let me know how the KIP looks now. Is it at a stage where I can
> start with the Voting phase? Of course I am still open to
> feedback/suggestions but planning to start the Vote for it.
>
> Thanks!
> Sagar.
>
> On Tue, Jul 11, 2023 at 10:00 PM Sagar  wrote:
>
> > Hi Yash/Chris,
> >
> > Thanks for the feedback! I have updated the KIP with the suggestions
> > provided. I would also update the PR with the suggestions.
> >
> > Also, I was hoping that this could make it to the 3.6 release given that
> > it would benefit source connectors which have some of the problems listed
> > in the Motivation Section.
> >
> > Responses Inline:
> >
> > Yash:
> >
> > 1) In the proposed changes section where you talk about modifying the
> >> offsets, could you please clarify that tasks shouldn't modify the
> offsets
> >> map that is passed as an argument? Currently, the distinction between
> the
> >> offsets map passed as an argument and the offsets map that is returned
> is
> >> not very clear in numerous places.
> >
> >
> >
> > Added
> >
> > 2) The default return value of Optional.empty() seems to be fairly
> >> non-intuitive considering that the return value is supposed to be the
> >> offsets that are to be committed. Can we consider simply returning the
> >> offsets argument itself by default instead?
> >
> >
> >
> > Chris is suggesting returning null for the default case. I am thinking to
> > make null
> > as the default return type. If the returned map is null, there won't be
> > any further
> > processing otherwise we will contonue with the existing logic.
> >
> > 3) The KIP states that "It is also possible that a task might choose to
> >> send a tombstone record as an offset. This is not recommended and to
> >> prevent connectors shooting themselves in the foot due to this" - could
> >> you
> >> please clarify why this is not recommended / supported?
> >
> >
> >
> > I have added that a better way of doing that would be via KIP-875. Also,
> I
> > didn't want to include
> > any mechamisms for users to meddle with the offsets topic. Allowing
> > tombstone records via this method
> > would be akin to publishing tombstone records directly to the offsets
> > topic which is not recommended
> > generally.
> >
> > 4) The KIP states that "If a task returns an Optional of a null object or
> >> an Optional of an empty map, even for such cases the behaviour would
> would
> >> be disabled." - since this is an optional API that source task
> >> implementations don't necessarily need to implement, I don't think I
> fully
> >> follow why the return type of the proposed "updateOffsets" method is an
> >> Optional? Can we not simply use the Map as the return type instead?
> >
> >
> >
> > Yeah, I updated the 

Re: [VOTE] KIP-930: Tiered Storage Metrics

[Abhijeet Kumar]

Please find the updated link to the KIP:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-930%3A+Rename+ambiguous+Tiered+Storage+Metrics

Updated the KIP as per our conversation on the discussion thread.

On Tue, Jul 25, 2023 at 11:29 AM Abhijeet Kumar 
wrote:

> Hi All,
>
> I would like to start the vote for KIP-930 Tiered Storage Metrics.
>
> The KIP is here:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-930%3A+Tiered+Storage+Metrics
>
> Regards
> Abhijeet.
>
>

-- 
Abhijeet.

Re: [DISCUSS] KIP-930: Tiered Storage Metrics

[Jorge Esteban Quilcate Otoya]

Hi Abhijeet,

Thanks for this KIP, I pretty much agree with the renaming and new names
look good to me.

Cheers,
Jorge.

On Tue, 25 Jul 2023 at 12:56, Satish Duggana 
wrote:

> Hi Abhijeet,
> Thanks for keeping this KIP only to renaming the existing metrics for
> better clarity. These new names look good to me.
>
> ~Satish.
>
> On Tue, 25 Jul 2023 at 13:12, Luke Chen  wrote:
> >
> > Hi Abhijeet,
> >
> > Thanks for the KIP!
> > I don't have much preference for the name changing.
> > But if it could confuse other people, it's good to make it clear.
> >
> > Thank you.
> > Luke
> >
> > On Tue, Jul 25, 2023 at 2:53 PM Abhijeet Kumar <
> abhijeet.cse@gmail.com>
> > wrote:
> >
> > > Hi Kamal,
> > >
> > > As we discussed offline, I will rename this KIP so that it only
> captures
> > > the aspect of renaming the previously added metrics to remove
> ambiguity.
> > > I will create another KIP for RemoteIndexCache metrics and other
> relevant
> > > tiered storage metrics.
> > >
> > > On Tue, Jul 25, 2023 at 12:03 PM Kamal Chandraprakash <
> > > kamal.chandraprak...@gmail.com> wrote:
> > >
> > > > Hi Abhijeet,
> > > >
> > > > Thanks for the KIP!
> > > >
> > > > We are changing the metric names from what was proposed in the
> KIP-405
> > > and
> > > > adding new metrics for RemoteIndexCache.
> > > > In the KIP, it's not clear whether we are renaming the aggregate
> broker
> > > > level metrics for remote copy/fetch/failed-copy/failed-fetch.
> > > >
> > > > Are these metrics enough to monitor all the aspects of tiered
> storage?
> > > >
> > > > (eg)
> > > > 1. Metrics to see the Tier Lag Status by number of pending
> > > > segments/records.
> > > > 2. Similar to log-start-offset and log-end-offset metrics.  Should we
> > > > expose local-log-start-offset and
> > > highest-offset-uploaded-to-remote-storage
> > > > as metric?
> > > >
> > > > Thanks,
> > > > Kamal
> > > >
> > > > On Mon, Jul 24, 2023 at 2:08 PM Abhijeet Kumar <
> > > abhijeet.cse@gmail.com
> > > > >
> > > > wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > I created KIP-930 for adding RemoteIndexCache stats and also to
> rename
> > > > some
> > > > > tiered storage metrics added as part of KIP-405
> > > > > <
> > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage#KIP405:KafkaTieredStorage-NewMetrics
> > > > > >
> > > > > to remove ambiguity.
> > > > >
> > > > >
> > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-930%3A+Tiered+Storage+Metrics
> > > > >
> > > > > Feedback and suggestions are welcome.
> > > > >
> > > > > Regards,
> > > > > Abhijeet.
> > > > >
> > > >
> > >
> > >
> > > --
> > > Abhijeet.
> > >
>

Re: Apache Kafka 3.6.0 release

[Mayank Shekhar Narula]

Hi Satish

Heads up KIP-951 is under voting. This could be delayed by a day or two for
the 3.6 KIP deadline.

On Tue, Jul 25, 2023 at 4:19 AM Satish Duggana 
wrote:

> Thanks Colin for the update on the mentioned KIPs.
>
> ~Satish.
>
> On Mon, 24 Jul 2023 at 23:09, Colin McCabe  wrote:
> >
> > Hi Satish,
> >
> > I removed "KIP-866 ZooKeeper to KRaft Migration" from the list of
> pending KIPs, since that one was shipped in 3.4. I added "KIP-868 Metadata
> Transactions", since we are planning on implementing this in 3.6. (The KIP
> was approved a while ago, but not yet shipped.)
> >
> > I also added "KIP-938: Add more metrics for measuring KRaft
> performance," which is a new KIP we are implemeting in 3.6 (The JIRA is
> open now for review.) Same for KIP-919 which is being voted on now.
> >
> > best,
> > Colin
> >
> > On Mon, Jul 24, 2023, at 03:59, Satish Duggana wrote:
> > > A gentle reminder on the KIP freeze date: 26th Jul. Please try to
> > > close discussion/vote threads asap.
> > >
> > > Thanks,
> > > Satish.
> > >
> > > On Sun, 23 Jul 2023 at 11:10, Satish Duggana 
> wrote:
> > >>
> > >> Thanks Colov/Divij for adding the KIP-952. I do not think it is a
> > >> blocker for 3.6.0. We can discuss the KIP in the respective thread.
> > >>
> > >> ~Satish.
> > >>
> > >> On Sun, 23 Jul 2023 at 07:21, Satish Duggana <
> satish.dugg...@gmail.com> wrote:
> > >> >
> > >> > Thanks ShunKang for the update. I added both the KIPs to the wiki.
> > >> > Please feel free to update the wiki with the latest.
> > >> >
> > >> > ~Satish.
> > >> >
> > >> > On Sat, 22 Jul 2023 at 22:50, ShunKang Lin <
> linshunkang@gmail.com> wrote:
> > >> > >
> > >> > > Hi Satish,
> > >> > >
> > >> > > Could we add "KIP-863: Reduce CompletedFetch#parseRecord() memory
> copy" [1]
> > >> > > and "KIP-872: Add Serializer#serializeToByteBuffer() to reduce
> memory
> > >> > > copying" [2] to the release plan?
> > >> > > Thanks!
> > >> > >
> > >> > > [1]
> > >> > >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=225152035
> > >> > >
> > >> > > [2]
> > >> > >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=228495828
> > >> > > I would appreciate a few more reviews on the pull request (
> > >> > > https://github.com/apache/kafka/pull/12685) for KIP-872.
> > >> > >
> > >> > > Best,
> > >> > > ShunKang
> > >> > >
> > >> > > Divij Vaidya  于2023年7月22日周六 20:06写道：
> > >> > >
> > >> > > > Hi Satish
> > >> > > >
> > >> > > > I have added the following accepted KIPs to the release plan.
> Please let me
> > >> > > > know if something requires a change.
> > >> > > >
> > >> > > > Accepted KIPs -
> > >> > > >
> > >> > > > 1.
> > >> > > >
> > >> > > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-852%3A+Optimize+calculation+of+size+for+log+in+remote+tier
> > >> > > >
> > >> > > > 2.
> > >> > > >
> > >> > > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-937%3A+Improve+Message+Timestamp+Validation
> > >> > > >
> > >> > > >
> > >> > > > Pending discussion KIP which I believe is important to be
> merged into 3.6 -
> > >> > > >
> > >> > > > 3.
> > >> > > >
> > >> > > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-952%3A+Regenerate+segment-aligned+producer+snapshots+when+upgrading+to+a+Kafka+version+supporting+Tiered+Storage
> > >> > > >
> > >> > > >
> > >> > > > --
> > >> > > > Divij Vaidya
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > On Sat, Jul 22, 2023 at 6:41 AM Satish Duggana <
> satish.dugg...@gmail.com>
> > >> > > > wrote:
> > >> > > >
> > >> > > > > Thanks Hao for the update on KIP-925.
> > >> > > > >
> > >> > > > > On Thu, 20 Jul 2023 at 23:05, Hao Li 
> wrote:
> > >> > > > > >
> > >> > > > > > Hi Satish,
> > >> > > > > >
> > >> > > > > > KIP-925 was accepted and currently under implementation. I
> just added
> > >> > > > it
> > >> > > > > to
> > >> > > > > > the release plan.
> > >> > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-925%3A+Rack+aware+task+assignment+in+Kafka+Streams
> > >> > > > > >
> > >> > > > > > Thanks,
> > >> > > > > > Hao
> > >> > > > > >
> > >> > > > > > On Thu, Jul 20, 2023 at 6:18 AM Christo Lolov <
> christolo...@gmail.com>
> > >> > > > > > wrote:
> > >> > > > > >
> > >> > > > > > > Hello!
> > >> > > > > > >
> > >> > > > > > > A couple of days ago I opened a new KIP for discussion -
> KIP-952
> > >> > > > [1]. I
> > >> > > > > > > believe it might be a blocker for the release of 3.6.0,
> but I wanted
> > >> > > > to
> > >> > > > > > > bring it up here for a decision on its urgency with the
> current set
> > >> > > > of
> > >> > > > > > > people who are looking at Tiered Storage (Satish, Luke,
> Ivan, Divij)
> > >> > > > > given
> > >> > > > > > > that the date for KIP freeze is fast approaching.
> > >> > > > > > > What are your thoughts on the matter?
> > >> > > > > > >
> > >> > > > > > > [1]
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > >
> > >> > > >
> 

Re: [DISCUSS] KIP-930: Tiered Storage Metrics

[Satish Duggana]

Hi Abhijeet,
Thanks for keeping this KIP only to renaming the existing metrics for
better clarity. These new names look good to me.

~Satish.

On Tue, 25 Jul 2023 at 13:12, Luke Chen  wrote:
>
> Hi Abhijeet,
>
> Thanks for the KIP!
> I don't have much preference for the name changing.
> But if it could confuse other people, it's good to make it clear.
>
> Thank you.
> Luke
>
> On Tue, Jul 25, 2023 at 2:53 PM Abhijeet Kumar 
> wrote:
>
> > Hi Kamal,
> >
> > As we discussed offline, I will rename this KIP so that it only captures
> > the aspect of renaming the previously added metrics to remove ambiguity.
> > I will create another KIP for RemoteIndexCache metrics and other relevant
> > tiered storage metrics.
> >
> > On Tue, Jul 25, 2023 at 12:03 PM Kamal Chandraprakash <
> > kamal.chandraprak...@gmail.com> wrote:
> >
> > > Hi Abhijeet,
> > >
> > > Thanks for the KIP!
> > >
> > > We are changing the metric names from what was proposed in the KIP-405
> > and
> > > adding new metrics for RemoteIndexCache.
> > > In the KIP, it's not clear whether we are renaming the aggregate broker
> > > level metrics for remote copy/fetch/failed-copy/failed-fetch.
> > >
> > > Are these metrics enough to monitor all the aspects of tiered storage?
> > >
> > > (eg)
> > > 1. Metrics to see the Tier Lag Status by number of pending
> > > segments/records.
> > > 2. Similar to log-start-offset and log-end-offset metrics.  Should we
> > > expose local-log-start-offset and
> > highest-offset-uploaded-to-remote-storage
> > > as metric?
> > >
> > > Thanks,
> > > Kamal
> > >
> > > On Mon, Jul 24, 2023 at 2:08 PM Abhijeet Kumar <
> > abhijeet.cse@gmail.com
> > > >
> > > wrote:
> > >
> > > > Hi All,
> > > >
> > > > I created KIP-930 for adding RemoteIndexCache stats and also to rename
> > > some
> > > > tiered storage metrics added as part of KIP-405
> > > > <
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage#KIP405:KafkaTieredStorage-NewMetrics
> > > > >
> > > > to remove ambiguity.
> > > >
> > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-930%3A+Tiered+Storage+Metrics
> > > >
> > > > Feedback and suggestions are welcome.
> > > >
> > > > Regards,
> > > > Abhijeet.
> > > >
> > >
> >
> >
> > --
> > Abhijeet.
> >

Re: [DISCUSS] KIP-953: partition method to be overloaded to accept headers as well.

[Sagar]

Hi Jack,

Thanks I have a couple of final comments and then I am good.

1) Can you elaborate on the Javadocs of the partition headers argument to
specify that a null headers value is equivalent to invoking the older
partition method? It is apparent but generally good to call out.
2) In the Compatibility section, you have mentioned backward comparable. I
believe it should be *backward compatible change.*

I don't have other comments. Post this, probably someone else who has more
context on Clients can also chime in on this before we can move this to
Voting.

Thanks!
Sagar.


On Sat, Jul 22, 2023 at 10:09 AM Jack Tomy  wrote:

> Hey @Sagar,
>
> Thank you again for the response and feedback.
>
>1. Though the ask wasn't very clear to me I have attached the Javadoc as
>per your suggestion. Please have a look and let me know if this meets
> the
>expectations.
>2. Done.
>3. Done
>4. Done
>
> Hey @Sagar and everyone,
> Please have a look at the new version and share your thoughts.
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263424937
>
> On Thu, Jul 20, 2023 at 9:46 PM Sagar  wrote:
>
> > Thanks Jack for the updates.
> >
> > Some more feedback:
> >
> > 1) It would be better if you can add the Javadoc in the Public interfaces
> > section. That is a general practice used which gives the readers of the
> KIP
> > a high level idea of the Public Interfaces.
> >
> > 2) In the proposed section, the bit about marking headers as read only
> > seems like an implementation detail This can generally be avoided in
> KIPs.
> >
> > 3) Also, in the Deprecation section, can you mention again that this is a
> > backward compatible change and the reason for it (already done in the
> > Proposed Changes section).
> >
> > 4) In the Testing Plan section, there is still the KIP template bit
> copied
> > over. That can be removed.
> >
> > Thanks!
> > Sagar.
> >
> >
> > On Thu, Jul 20, 2023 at 2:48 PM Jack Tomy  wrote:
> >
> > > Hey Everyone,
> > >
> > > Please consider this as a reminder and share your feedback. Thank you.
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263424937
> > >
> > > On Tue, Jul 18, 2023 at 5:43 PM Jack Tomy 
> wrote:
> > >
> > > > Hey @Sagar,
> > > >
> > > > Thank you for the response and feedback.
> > > >
> > > >1. Done
> > > >2. Yeah, that was a mistake from my end. Corrected.
> > > >3. Can you please elaborate this, I have added the java doc along
> > with
> > > >the code changes. Should I paste the same in KIP too?
> > > >4. Moved.
> > > >5. I have added one more use case, it is actually helpful in any
> > > >situation where you want to pass some information to partition
> > method
> > > but
> > > >don't have to have it in the key or value.
> > > >6. Added.
> > > >
> > > >
> > > > Hey @Sagar and everyone,
> > > > Please have a look at the new version and share your thoughts.
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263424937
> > > >
> > > >
> > > > On Tue, Jul 18, 2023 at 9:53 AM Sagar 
> > wrote:
> > > >
> > > >> Hi Jack,
> > > >>
> > > >> Thanks for the KIP! Seems like an interesting idea. I have some
> > > feedback:
> > > >>
> > > >> 1) It would be great if you could clean up the text that seems to
> > mimic
> > > >> the
> > > >> KIP template. It is generally not required in the KIP.
> > > >>
> > > >> 2) In the Public Interfaces where you mentioned *Partitioner method
> in
> > > >> **org/apache/kafka/clients/producer
> > > >> will have the following update*, I believe you meant the Partitioner
> > > >> *interface*?
> > > >>
> > > >> 3) Staying on Public Interface, it is generally preferable to add a
> > > >> Javadocs section along with the newly added method. You could also
> > > >> describe
> > > >> the behaviour of it invoking the default existing method.
> > > >>
> > > >> 4) The option that is mentioned in the Rejected Alternatives, seems
> > more
> > > >> like a workaround to the current problem that you are describing.
> That
> > > >> could be added to the Motivation section IMO.
> > > >>
> > > >> 5) Can you also add some more examples of scenarios where this would
> > be
> > > >> helpful? The only scenario mentioned seems to have a workaround.
> Just
> > > >> trying to ensure that we have a strong enough motivation before
> > adding a
> > > >> public API.
> > > >>
> > > >> 6) One thing which should also be worth noting down would be what
> > > happens
> > > >> if users override both methods, only one method (new or old) and no
> > > >> methods
> > > >> (the default behaviour). It would help in understanding the proposal
> > > >> better.
> > > >>
> > > >> Thanks!
> > > >> Sagar.
> > > >>
> > > >>
> > > >> On Mon, Jul 17, 2023 at 9:19 PM Jack Tomy 
> > > wrote:
> > > >>
> > > >> > Hey everyone,
> > > >> >
> > > >> > Not seeing much discussion on the KPI. Might be because it is too
> > > >> > obvious .
> > > >> >
> > > >> > If there are no more 

Question about Lag Calculations in Apache Kafka Source Code

[Henry GALVEZ]

Hi everyone,

I am interested in understanding how the broker performs lag calculations. 
Specifically, I would like to explore the possibility of improving the 
calculation method to use the latest stable offset instead of the latest offset.

I noticed that there might be differences between the results obtained from 
AdminClient.listOffsets and AdminClient.listConsumerGroupOffsets, and I believe 
investigating this area in the source code might shed some light on potential 
optimizations.

Could you please guide me to the specific part of the Apache Kafka source code 
where the lag calculations are performed? I would greatly appreciate any 
insights or pointers you can provide to help me get started with my 
investigation.

Thank you in advance for your assistance. Looking forward to hearing from you.

Best regards,
Henry

Re:Request permission to contribute

[Taras Ledkov]

Hi Guozhang, 

Thanks for your attention.

I'm a contributor of other Apache project (Ignite).
Now I don't have permissions to create a page in `Apache Kafka` space on wiki 
(confluence) [1] 

[1] 
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals

Re: [DISCUSS] KIP-930: Tiered Storage Metrics

[Luke Chen]

Hi Abhijeet,

Thanks for the KIP!
I don't have much preference for the name changing.
But if it could confuse other people, it's good to make it clear.

Thank you.
Luke

On Tue, Jul 25, 2023 at 2:53 PM Abhijeet Kumar 
wrote:

> Hi Kamal,
>
> As we discussed offline, I will rename this KIP so that it only captures
> the aspect of renaming the previously added metrics to remove ambiguity.
> I will create another KIP for RemoteIndexCache metrics and other relevant
> tiered storage metrics.
>
> On Tue, Jul 25, 2023 at 12:03 PM Kamal Chandraprakash <
> kamal.chandraprak...@gmail.com> wrote:
>
> > Hi Abhijeet,
> >
> > Thanks for the KIP!
> >
> > We are changing the metric names from what was proposed in the KIP-405
> and
> > adding new metrics for RemoteIndexCache.
> > In the KIP, it's not clear whether we are renaming the aggregate broker
> > level metrics for remote copy/fetch/failed-copy/failed-fetch.
> >
> > Are these metrics enough to monitor all the aspects of tiered storage?
> >
> > (eg)
> > 1. Metrics to see the Tier Lag Status by number of pending
> > segments/records.
> > 2. Similar to log-start-offset and log-end-offset metrics.  Should we
> > expose local-log-start-offset and
> highest-offset-uploaded-to-remote-storage
> > as metric?
> >
> > Thanks,
> > Kamal
> >
> > On Mon, Jul 24, 2023 at 2:08 PM Abhijeet Kumar <
> abhijeet.cse@gmail.com
> > >
> > wrote:
> >
> > > Hi All,
> > >
> > > I created KIP-930 for adding RemoteIndexCache stats and also to rename
> > some
> > > tiered storage metrics added as part of KIP-405
> > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage#KIP405:KafkaTieredStorage-NewMetrics
> > > >
> > > to remove ambiguity.
> > >
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-930%3A+Tiered+Storage+Metrics
> > >
> > > Feedback and suggestions are welcome.
> > >
> > > Regards,
> > > Abhijeet.
> > >
> >
>
>
> --
> Abhijeet.
>

[jira] [Created] (KAFKA-15246) CoordinatorContext should be protected by a lock

[David Jacot (Jira)]

David Jacot created KAFKA-15246:
---

 Summary: CoordinatorContext should be protected by a lock
 Key: KAFKA-15246
 URL: https://issues.apache.org/jira/browse/KAFKA-15246
 Project: Kafka
  Issue Type: Sub-task
Reporter: David Jacot
Assignee: David Jacot






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #2034

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-930: Tiered Storage Metrics

[Abhijeet Kumar]

Hi Kamal,

As we discussed offline, I will rename this KIP so that it only captures
the aspect of renaming the previously added metrics to remove ambiguity.
I will create another KIP for RemoteIndexCache metrics and other relevant
tiered storage metrics.

On Tue, Jul 25, 2023 at 12:03 PM Kamal Chandraprakash <
kamal.chandraprak...@gmail.com> wrote:

> Hi Abhijeet,
>
> Thanks for the KIP!
>
> We are changing the metric names from what was proposed in the KIP-405 and
> adding new metrics for RemoteIndexCache.
> In the KIP, it's not clear whether we are renaming the aggregate broker
> level metrics for remote copy/fetch/failed-copy/failed-fetch.
>
> Are these metrics enough to monitor all the aspects of tiered storage?
>
> (eg)
> 1. Metrics to see the Tier Lag Status by number of pending
> segments/records.
> 2. Similar to log-start-offset and log-end-offset metrics.  Should we
> expose local-log-start-offset and highest-offset-uploaded-to-remote-storage
> as metric?
>
> Thanks,
> Kamal
>
> On Mon, Jul 24, 2023 at 2:08 PM Abhijeet Kumar  >
> wrote:
>
> > Hi All,
> >
> > I created KIP-930 for adding RemoteIndexCache stats and also to rename
> some
> > tiered storage metrics added as part of KIP-405
> > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage#KIP405:KafkaTieredStorage-NewMetrics
> > >
> > to remove ambiguity.
> >
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-930%3A+Tiered+Storage+Metrics
> >
> > Feedback and suggestions are welcome.
> >
> > Regards,
> > Abhijeet.
> >
>


-- 
Abhijeet.

Re: [DISCUSS] KIP-930: Tiered Storage Metrics

[Kamal Chandraprakash]

Hi Abhijeet,

Thanks for the KIP!

We are changing the metric names from what was proposed in the KIP-405 and
adding new metrics for RemoteIndexCache.
In the KIP, it's not clear whether we are renaming the aggregate broker
level metrics for remote copy/fetch/failed-copy/failed-fetch.

Are these metrics enough to monitor all the aspects of tiered storage?

(eg)
1. Metrics to see the Tier Lag Status by number of pending segments/records.
2. Similar to log-start-offset and log-end-offset metrics.  Should we
expose local-log-start-offset and highest-offset-uploaded-to-remote-storage
as metric?

Thanks,
Kamal

On Mon, Jul 24, 2023 at 2:08 PM Abhijeet Kumar 
wrote:

> Hi All,
>
> I created KIP-930 for adding RemoteIndexCache stats and also to rename some
> tiered storage metrics added as part of KIP-405
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage#KIP405:KafkaTieredStorage-NewMetrics
> >
> to remove ambiguity.
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-930%3A+Tiered+Storage+Metrics
>
> Feedback and suggestions are welcome.
>
> Regards,
> Abhijeet.
>

[VOTE] KIP-930: Tiered Storage Metrics

[Abhijeet Kumar]

Hi All,

I would like to start the vote for KIP-930 Tiered Storage Metrics.

The KIP is here:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-930%3A+Tiered+Storage+Metrics

Regards
Abhijeet.