Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1990

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 393521 lines...]

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
StoreQueryIntegrationTest > shouldQueryOnlyActivePartitionStoresByDefault() 
PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
StoreQueryIntegrationTest > 
shouldQueryStoresAfterAddingAndRemovingStreamThread() STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
StoreQueryIntegrationTest > 
shouldQueryStoresAfterAddingAndRemovingStreamThread() PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
StoreQueryIntegrationTest > 
shouldQuerySpecificStalePartitionStoresMultiStreamThreadsNamedTopology() STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
StoreQueryIntegrationTest > 
shouldQuerySpecificStalePartitionStoresMultiStreamThreadsNamedTopology() PASSED
streams-3: SMOKE-TEST-CLIENT-CLOSED
streams-6: SMOKE-TEST-CLIENT-CLOSED
streams-2: SMOKE-TEST-CLIENT-CLOSED
streams-3: SMOKE-TEST-CLIENT-CLOSED
streams-4: SMOKE-TEST-CLIENT-CLOSED
streams-0: SMOKE-TEST-CLIENT-CLOSED
streams-5: SMOKE-TEST-CLIENT-CLOSED
streams-8: SMOKE-TEST-CLIENT-CLOSED
streams-4: SMOKE-TEST-CLIENT-CLOSED
streams-1: SMOKE-TEST-CLIENT-CLOSED
streams-2: SMOKE-TEST-CLIENT-CLOSED
streams-5: SMOKE-TEST-CLIENT-CLOSED
streams-1: SMOKE-TEST-CLIENT-CLOSED
streams-6: SMOKE-TEST-CLIENT-CLOSED
streams-0: SMOKE-TEST-CLIENT-CLOSED
streams-7: SMOKE-TEST-CLIENT-CLOSED
streams-7: SMOKE-TEST-CLIENT-CLOSED

Deprecated Gradle features were used in this build, making it incompatible with 
Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings 
and determine if they come from your own scripts or plugins.

See 
https://docs.gradle.org/8.1.1/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 3h 5m 11s
231 actionable tasks: 124 executed, 107 up-to-date

See the profiling report at: 
file:///home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk_2/build/reports/profile/profile-2023-07-11-22-31-17.html
A fine-grained performance profile is available: use the --scan option.
[Pipeline] junit
Recording test results
[Checks API] No suitable checks publisher found.
[Pipeline] echo
Skipping Kafka Streams archetype test for Java 11
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // stage
[Pipeline] }

Gradle Test Run :streams:integrationTest > Gradle Test Executor 183 > 
HighAvailabilityTaskAssignorIntegrationTest > 
shouldScaleOutWithWarmupTasksAndPersistentStores(TestInfo) PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 183 > 
HighAvailabilityTaskAssignorIntegrationTest > 
shouldScaleOutWithWarmupTasksAndInMemoryStores(TestInfo) STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 183 > 
HighAvailabilityTaskAssignorIntegrationTest > 
shouldScaleOutWithWarmupTasksAndInMemoryStores(TestInfo) PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 183 > 
KStreamAggregationDedupIntegrationTest > shouldReduce(TestInfo) STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 183 > 
KStreamAggregationDedupIntegrationTest > shouldReduce(TestInfo) PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 183 > 
KStreamAggregationDedupIntegrationTest > shouldGroupByKey(TestInfo) STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 183 > 
KStreamAggregationDedupIntegrationTest > shouldGroupByKey(TestInfo) PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 183 > 
KStreamAggregationDedupIntegrationTest > shouldReduceWindowed(TestInfo) STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 183 > 
KStreamAggregationDedupIntegrationTest > shouldReduceWindowed(TestInfo) PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 183 > 
KStreamKStreamIntegrationTest > shouldOuterJoin() STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 183 > 
KStreamKStreamIntegrationTest > shouldOuterJoin() PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 183 > 
KTableSourceTopicRestartIntegrationTest > 
shouldRestoreAndProgressWhenTopicNotWrittenToDuringRestoration() STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 183 > 
KTableSourceTopicRestartIntegrationTest > 
shouldRestoreAndProgressWhenTopicNotWrittenToDuringRestoration() PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 183 > 
KTableSourceTopicRestartIntegrationTest > 
shouldRestoreAndProgressWhenTopicWrittenToDuringRestorationWithEosAlphaEnabled()
 STARTED

Gradle Test Run :streams:integrationTest 

Jenkins build is unstable: Kafka » Kafka Branch Builder » 3.5 #36

[Apache Jenkins Server]

See 

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1989

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 386521 lines...]
> Task :connect:json:compileTestJava UP-TO-DATE
> Task :connect:api:compileTestJava UP-TO-DATE
> Task :connect:api:testClasses UP-TO-DATE
> Task :connect:json:copyDependantLibs UP-TO-DATE
> Task :connect:json:jar UP-TO-DATE
> Task :connect:json:generateMetadataFileForMavenJavaPublication
> Task :connect:json:testClasses UP-TO-DATE
> Task :connect:json:testJar
> Task :connect:api:testJar
> Task :connect:json:testSrcJar
> Task :clients:generateMetadataFileForMavenJavaPublication
> Task :connect:api:testSrcJar
> Task :storage:api:compileTestJava
> Task :storage:api:testClasses
> Task :connect:json:publishMavenJavaPublicationToMavenLocal
> Task :connect:json:publishToMavenLocal
> Task :connect:api:publishMavenJavaPublicationToMavenLocal
> Task :connect:api:publishToMavenLocal
> Task :server-common:compileTestJava
> Task :server-common:testClasses
> Task :raft:compileTestJava
> Task :raft:testClasses
> Task :group-coordinator:compileTestJava
> Task :group-coordinator:testClasses

> Task :clients:javadoc
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/clients/src/main/java/org/apache/kafka/common/config/ConfigDef.java:81:
 warning - Tag @link:illegal character: "60" in "#define(String, Type, 
Importance, String, String, int, Width, String, List)"
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/clients/src/main/java/org/apache/kafka/common/config/ConfigDef.java:81:
 warning - Tag @link:illegal character: "62" in "#define(String, Type, 
Importance, String, String, int, Width, String, List)"
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/clients/src/main/java/org/apache/kafka/common/config/ConfigDef.java:81:
 warning - Tag @link: can't find define(String, Type, Importance, String, 
String, int, Width, String, List) in 
org.apache.kafka.common.config.ConfigDef
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/clients/src/main/java/org/apache/kafka/clients/admin/ScramMechanism.java:32:
 warning - Tag @see: missing final '>': "https://cwiki.apache.org/confluence/display/KAFKA/KIP-554%3A+Add+Broker-side+SCRAM+Config+API;>KIP-554:
 Add Broker-side SCRAM Config API

 This code is duplicated in 
org.apache.kafka.common.security.scram.internals.ScramMechanism.
 The type field in both files must match and must not change. The type field
 is used both for passing ScramCredentialUpsertion and for the internal
 UserScramCredentialRecord. Do not change the type field."
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/secured/package-info.java:21:
 warning - Tag @link: reference not found: 
org.apache.kafka.common.security.oauthbearer

> Task :metadata:compileTestJava
> Task :metadata:testClasses

> Task :clients:javadoc
5 warnings

> Task :clients:javadocJar
> Task :clients:srcJar
> Task :clients:testJar
> Task :clients:testSrcJar
> Task :clients:publishMavenJavaPublicationToMavenLocal
> Task :clients:publishToMavenLocal
> Task :core:classes
> Task :core:compileTestJava NO-SOURCE
> Task :core:compileTestScala
> Task :core:testClasses
> Task :streams:compileTestJava
> Task :streams:testClasses
> Task :streams:testJar
> Task :streams:testSrcJar
> Task :streams:publishMavenJavaPublicationToMavenLocal
> Task :streams:publishToMavenLocal

Deprecated Gradle features were used in this build, making it incompatible with 
Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings 
and determine if they come from your own scripts or plugins.

See 
https://docs.gradle.org/8.1.1/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 5m 21s
89 actionable tasks: 41 executed, 48 up-to-date
[Pipeline] sh
+ grep ^version= gradle.properties
+ cut -d= -f 2
[Pipeline] dir
Running in 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/quickstart
[Pipeline] {
[Pipeline] sh
+ mvn clean install -Dgpg.skip
[INFO] Scanning for projects...
[INFO] 
[INFO] Reactor Build Order:
[INFO] 
[INFO] Kafka Streams :: Quickstart[pom]
[INFO] streams-quickstart-java[maven-archetype]
[INFO] 
[INFO] < org.apache.kafka:streams-quickstart >-
[INFO] Building Kafka Streams :: Quickstart 3.6.0-SNAPSHOT[1/2]
[INFO]   from pom.xml
[INFO] [ pom ]-
[INFO] 
[INFO] --- clean:3.0.0:clean (default-clean) @ streams-quickstart ---
[INFO] 
[INFO] --- remote-resources:1.5:process (process-resource-bundles) @ 
streams-quickstart ---
[INFO] 
[INFO] --- site:3.5.1:attach-descriptor (attach-descriptor) @ 
streams-quickstart ---
[INFO] 
[INFO] --- gpg:1.6:sign (sign-artifacts) @ 

Re: [DISCUSS] KIP-916: MM2 distributed mode flow log context

[Greg Harris]

Hey Daniel,

Thanks for the KIP! The current logging behavior seems like a
usability nightmare. I've only debugged single-worker JVM deployments,
and I see now that I took the uniqueness of a connector name for
granted.

I had a few questions about the change:

1. The MDC context change and the thread name change both seem to
solve the problem, and I assume that the majority of people would find
the thread name sufficient and leave things at their default. What is
the advantage of implementing both vs picking one?

2. I see that there is already a `workerId` that is passed around for
emitting to the status topic and to namespace the metrics. In MM2
dedicated mode, this contains the source and target. Do you think it
could be used to disambiguate the log context for
WorkerConnector/WorkerSinkTask as well?

3. Is the 'flow' (source->target) always more relevant than the
'source' and 'target' separately? Does it ever make sense to aggregate
logs for one source and multiple targets, or vice versa, and should we
expose these fields separately?

Overall, I think this feels like a shared-JVM problem that just
happens to appear in MM2 dedicated mode. I am interested to see if
there is a solution to this problem that also makes debugging easier
for other shared-JVM use-cases. We've also solved it in a number of
different ways and I wonder if there's a holistic way to namespace the
logging, metrics, and threads for workers in a shared JVM.

Thanks!
Greg

[jira] [Created] (KAFKA-15180) Generalize integration tests to change use of KafkaConsumer to Consumer

[Kirk True (Jira)]

Kirk True created KAFKA-15180:
-

 Summary: Generalize integration tests to change use of 
KafkaConsumer to Consumer
 Key: KAFKA-15180
 URL: https://issues.apache.org/jira/browse/KAFKA-15180
 Project: Kafka
  Issue Type: Test
  Components: clients, consumer
Reporter: Kirk True
Assignee: Kirk True


For the consumer threading refactor project, we're introducing a new 
implementation of the `Consumer` interface. However, most of the instances in 
the integration tests specifically use the concrete implementation 
`KafkaConsumer`. This task is to generalize those uses where possible to the 
`Consumer` interface.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.5 #35

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 563416 lines...]
Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
VersionedKeyValueStoreIntegrationTest > shouldSetChangelogTopicProperties PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
VersionedKeyValueStoreIntegrationTest > shouldRestore STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
VersionedKeyValueStoreIntegrationTest > shouldRestore PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
VersionedKeyValueStoreIntegrationTest > shouldPutGetAndDelete STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
VersionedKeyValueStoreIntegrationTest > shouldPutGetAndDelete PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
VersionedKeyValueStoreIntegrationTest > 
shouldManualUpgradeFromNonVersionedTimestampedToVersioned STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
VersionedKeyValueStoreIntegrationTest > 
shouldManualUpgradeFromNonVersionedTimestampedToVersioned PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
HandlingSourceTopicDeletionIntegrationTest > 
shouldThrowErrorAfterSourceTopicDeleted STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
HandlingSourceTopicDeletionIntegrationTest > 
shouldThrowErrorAfterSourceTopicDeleted PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
StreamsAssignmentScaleTest > testHighAvailabilityTaskAssignorLargeNumConsumers 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
StreamsAssignmentScaleTest > testHighAvailabilityTaskAssignorLargeNumConsumers 
PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorLargePartitionCount STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorLargePartitionCount PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorManyThreadsPerClient STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorManyThreadsPerClient PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
StreamsAssignmentScaleTest > testStickyTaskAssignorManyStandbys STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
StreamsAssignmentScaleTest > testStickyTaskAssignorManyStandbys PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
StreamsAssignmentScaleTest > testStickyTaskAssignorManyThreadsPerClient STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
StreamsAssignmentScaleTest > testStickyTaskAssignorManyThreadsPerClient PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
StreamsAssignmentScaleTest > testFallbackPriorTaskAssignorManyThreadsPerClient 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
StreamsAssignmentScaleTest > testFallbackPriorTaskAssignorManyThreadsPerClient 
PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
StreamsAssignmentScaleTest > testFallbackPriorTaskAssignorLargePartitionCount 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
StreamsAssignmentScaleTest > testFallbackPriorTaskAssignorLargePartitionCount 
PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
StreamsAssignmentScaleTest > testStickyTaskAssignorLargePartitionCount STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
StreamsAssignmentScaleTest > testStickyTaskAssignorLargePartitionCount PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
StreamsAssignmentScaleTest > testFallbackPriorTaskAssignorManyStandbys STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
StreamsAssignmentScaleTest > testFallbackPriorTaskAssignorManyStandbys PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
StreamsAssignmentScaleTest > testHighAvailabilityTaskAssignorManyStandbys 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
StreamsAssignmentScaleTest > testHighAvailabilityTaskAssignorManyStandbys PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
StreamsAssignmentScaleTest > testFallbackPriorTaskAssignorLargeNumConsumers 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
StreamsAssignmentScaleTest > testFallbackPriorTaskAssignorLargeNumConsumers 
PASSED

Gradle Test Run 

[GitHub] [kafka-site] stevenbooke commented on pull request #521: KAFKA-14995: Automate asf.yaml collaborators refresh

[via GitHub]

stevenbooke commented on PR #521:
URL: https://github.com/apache/kafka-site/pull/521#issuecomment-163185

   @mimaison I have hidden the GitHub usernames of the kafka committers, they 
do not get displayed when the website is rendered. The display is now the same 
as today.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [VOTE] KIP-944 Support async runtimes in consumer, votes needed!

[Philip Nee]

Hey Erik - Another thing I want to add to my comment is.  We are in-process
of re-writing the KafkaConsumer, and I think your proposal would work in
the new consumer because we are going to separate the user thread and the
background thread.  Here is the 1-pager, and we are in process of
converting this in to KIP-945.

Thanks,
P

On Tue, Jul 11, 2023 at 10:33 AM Philip Nee  wrote:

> Hey Erik,
>
> Sorry for holding up this email for a few days since Colin's response
> includes some of my concerns.  I'm in favor of this KIP, and I think your
> approach seems safe.  Of course, I probably missed something therefore I
> think this KIP needs to cover different use cases to demonstrate it doesn't
> cause any unsafe access. I think this can be demonstrated via diagrams and
> some code in the KIP.
>
> Thanks,
> P
>
> On Sat, Jul 8, 2023 at 12:28 PM Erik van Oosten
>  wrote:
>
>> Hello Colin,
>>
>>  >> In KIP-944, the callback thread can only delegate to another thread
>> after reading from and writing to a threadlocal variable, providing the
>> barriers right there.
>>
>>  > I don't see any documentation that accessing thread local variables
>> provides a total store or load barrier. Do you have such documentation?
>> It seems like if this were the case, we could eliminate volatile
>> variables from most of the code base.
>>
>> Now I was imprecise. The thread-locals are only somewhat involved. In
>> the KIP proposal the callback thread reads an access key from a
>> thread-local variable. It then needs to pass that access key to another
>> thread, which then can set it on its own thread-local variable. The act
>> of passing a value from one thread to another implies that a memory
>> barrier needs to be passed. However, this is all not so relevant since
>> there is no need to pass the access key back when the other thread is
>> done.
>>
>> But now I think about it a bit more, the locking mechanism runs in a
>> synchronized block. If I remember correctly this should be enough to
>> pass read and write barriers.
>>
>>  >> In the current implementation the consumer is also invoked from
>> random threads. If it works now, it should continue to work.
>>  > I'm not sure what you're referring to. Can you expand on this?
>>
>> Any invocation of the consumer (e.g. method poll) is not from a thread
>> managed by the consumer. This is what I was assuming you meant with the
>> term 'random thread'.
>>
>>  > Hmm, not sure what you mean by "cooperate with blocking code." If you
>> have 10 green threads you're multiplexing on to one CPU thread, and that
>> CPU thread gets blocked because of what one green thread is doing, the
>> other 9 green threads are blocked too, right? I guess it's "just" a
>> performance problem, but it still seems like it could be a serious one.
>>
>> There are several ways to deal with this. All async runtimes I know
>> (Akka, Zio, Cats-effects) support this by letting you mark a task as
>> blocking. The runtime will then either schedule it to another
>> thread-pool, or it will grow the thread-pool to accommodate. In any case
>> 'the other 9 green threads' will simply be scheduled to another real
>> thread. In addition, some of these runtimes detect long running tasks
>> and will reschedule waiting tasks to another thread. This is all a bit
>> off topic though.
>>
>>  > I don't see why this has to be "inherently multi-threaded." Why can't
>> we have the other threads report back what messages they've processed to
>> the worker thread. Then it will be able to handle these callbacks
>> without involving the other threads.
>>
>> Please consider the context which is that we are running inside the
>> callback of the rebalance listener. The only way to execute something
>> and also have a timeout on it is to run the something on another thread.
>>
>> Kind regards,
>>  Erik.
>>
>>
>> Op 08-07-2023 om 19:17 schreef Colin McCabe:
>> > On Sat, Jul 8, 2023, at 02:41, Erik van Oosten wrote:
>> >> Hi Colin,
>> >>
>> >> Thanks for your thoughts and taking the time to reply.
>> >>
>> >> Let me take away your concerns. None of your worries are an issue with
>> >> the algorithm described in KIP-944. Here it goes:
>> >>
>> >>   > It's not clear ot me that it's safe to access the Kafka consumer or
>> >>> producer concurrently from different threads.
>> >> Concurrent access is /not/ a design goal of KIP-944. In fact, it goes
>> >> through great lengths to make sure that this cannot happen.
>> >>
>> >> *The only design goal is to allow callbacks to call the consumer from
>> >> another thread.*
>> >>
>> >> To make sure there are no more misunderstandings about this, I have
>> >> added this goal to the KIP.
>> >>
>> > Hi Erik,
>> >
>> > Sorry, I spoke imprecisely. My concern is not concurrent access, but
>> multithreaded access in general. Basically cache line visibility issues.
>> >
>> >>   > This is true even if the accesses happen at different times,
>> because
>> >>> modern CPUs require memory barriers to guarantee 

[VOTE] KIP-759: Unneeded repartition canceling

[Shay Lin]

Hi all,

I'd like to call a vote on KIP-759: Unneeded repartition canceling
The KIP has been under discussion for quite some time(two years). This is a
valuable optimization for advanced users. I hope we can push this toward
the finish line this time.

Link to the KIP:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-759%3A+Unneeded+repartition+canceling

Best,
Shay

Re: [ANNOUNCE] New committer: Greg Harris

[Anna McDonald]

Congratulations Greg

On Tue, Jul 11, 2023 at 9:02 AM Jorge Esteban Quilcate Otoya <
quilcate.jo...@gmail.com> wrote:

> Congrats Greg!!
>
> On Tue 11. Jul 2023 at 15.20, Federico Valeri 
> wrote:
>
> > Congrats Greg!
> >
> > On Tue, Jul 11, 2023 at 3:55 AM Luke Chen  wrote:
> > >
> > > Congrats Greg!
> > >
> > > Luke
> > >
> > > On Tue, Jul 11, 2023 at 8:19 AM Matthew de Detrich
> > >  wrote:
> > >
> > > > Congratulations, well deserved!
> > > >
> > > > On Mon, Jul 10, 2023 at 5:45 PM Chris Egerton 
> > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > The PMC for Apache Kafka has invited Greg Harris to become a
> > committer,
> > > > and
> > > > > we are happy to announce that he has accepted!
> > > > >
> > > > > Greg has been contributing to Kafka since 2019. He has made over 50
> > > > commits
> > > > > mostly around Kafka Connect and Mirror Maker 2. His most notable
> > > > > contributions include KIP-898: "Modernize Connect plugin discovery"
> > and a
> > > > > deep overhaul of the offset syncing logic in MM2 that addressed
> > several
> > > > > technically-difficult, long-standing, high-impact issues.
> > > > >
> > > > > He has also been an active participant in discussions and reviews
> on
> > the
> > > > > mailing lists and on GitHub.
> > > > >
> > > > > Thanks for all of your contributions, Greg. Congratulations!
> > > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Matthew de Detrich
> > > >
> > > > *Aiven Deutschland GmbH*
> > > >
> > > > Immanuelkirchstraße 26, 10405 Berlin
> > > >
> > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > >
> > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > >
> > > > *m:* +491603708037
> > > >
> > > > *w:* aiven.io *e:* matthew.dedetr...@aiven.io
> > > >
> >
>

Re: [VOTE] KIP-944 Support async runtimes in consumer, votes needed!

[Philip Nee]

Hey Erik,

Sorry for holding up this email for a few days since Colin's response
includes some of my concerns.  I'm in favor of this KIP, and I think your
approach seems safe.  Of course, I probably missed something therefore I
think this KIP needs to cover different use cases to demonstrate it doesn't
cause any unsafe access. I think this can be demonstrated via diagrams and
some code in the KIP.

Thanks,
P

On Sat, Jul 8, 2023 at 12:28 PM Erik van Oosten
 wrote:

> Hello Colin,
>
>  >> In KIP-944, the callback thread can only delegate to another thread
> after reading from and writing to a threadlocal variable, providing the
> barriers right there.
>
>  > I don't see any documentation that accessing thread local variables
> provides a total store or load barrier. Do you have such documentation?
> It seems like if this were the case, we could eliminate volatile
> variables from most of the code base.
>
> Now I was imprecise. The thread-locals are only somewhat involved. In
> the KIP proposal the callback thread reads an access key from a
> thread-local variable. It then needs to pass that access key to another
> thread, which then can set it on its own thread-local variable. The act
> of passing a value from one thread to another implies that a memory
> barrier needs to be passed. However, this is all not so relevant since
> there is no need to pass the access key back when the other thread is done.
>
> But now I think about it a bit more, the locking mechanism runs in a
> synchronized block. If I remember correctly this should be enough to
> pass read and write barriers.
>
>  >> In the current implementation the consumer is also invoked from
> random threads. If it works now, it should continue to work.
>  > I'm not sure what you're referring to. Can you expand on this?
>
> Any invocation of the consumer (e.g. method poll) is not from a thread
> managed by the consumer. This is what I was assuming you meant with the
> term 'random thread'.
>
>  > Hmm, not sure what you mean by "cooperate with blocking code." If you
> have 10 green threads you're multiplexing on to one CPU thread, and that
> CPU thread gets blocked because of what one green thread is doing, the
> other 9 green threads are blocked too, right? I guess it's "just" a
> performance problem, but it still seems like it could be a serious one.
>
> There are several ways to deal with this. All async runtimes I know
> (Akka, Zio, Cats-effects) support this by letting you mark a task as
> blocking. The runtime will then either schedule it to another
> thread-pool, or it will grow the thread-pool to accommodate. In any case
> 'the other 9 green threads' will simply be scheduled to another real
> thread. In addition, some of these runtimes detect long running tasks
> and will reschedule waiting tasks to another thread. This is all a bit
> off topic though.
>
>  > I don't see why this has to be "inherently multi-threaded." Why can't
> we have the other threads report back what messages they've processed to
> the worker thread. Then it will be able to handle these callbacks
> without involving the other threads.
>
> Please consider the context which is that we are running inside the
> callback of the rebalance listener. The only way to execute something
> and also have a timeout on it is to run the something on another thread.
>
> Kind regards,
>  Erik.
>
>
> Op 08-07-2023 om 19:17 schreef Colin McCabe:
> > On Sat, Jul 8, 2023, at 02:41, Erik van Oosten wrote:
> >> Hi Colin,
> >>
> >> Thanks for your thoughts and taking the time to reply.
> >>
> >> Let me take away your concerns. None of your worries are an issue with
> >> the algorithm described in KIP-944. Here it goes:
> >>
> >>   > It's not clear ot me that it's safe to access the Kafka consumer or
> >>> producer concurrently from different threads.
> >> Concurrent access is /not/ a design goal of KIP-944. In fact, it goes
> >> through great lengths to make sure that this cannot happen.
> >>
> >> *The only design goal is to allow callbacks to call the consumer from
> >> another thread.*
> >>
> >> To make sure there are no more misunderstandings about this, I have
> >> added this goal to the KIP.
> >>
> > Hi Erik,
> >
> > Sorry, I spoke imprecisely. My concern is not concurrent access, but
> multithreaded access in general. Basically cache line visibility issues.
> >
> >>   > This is true even if the accesses happen at different times, because
> >>> modern CPUs require memory barriers to guarantee inter-thread visibilty
> >>> of loads and stores.
> >> In KIP-944, the callback thread can only delegate to another thread
> >> after reading from and writing to a threadlocal variable, providing the
> >> barriers right there.
> >>
> > I don't see any documentation that accessing thread local variables
> provides a total store or load barrier. Do you have such documentation? It
> seems like if this were the case, we could eliminate volatile variables
> from most of the code base.
> >
> >>   > I know 

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1988

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 387149 lines...]
Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testOuterWithLeftVersionedOnly[caching enabled = false] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testLeftWithRightVersionedOnly[caching enabled = false] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testLeftWithRightVersionedOnly[caching enabled = false] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testInnerInner[caching enabled = false] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testInnerInner[caching enabled = false] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testInnerOuter[caching enabled = false] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testInnerOuter[caching enabled = false] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testInnerLeft[caching enabled = false] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testInnerLeft[caching enabled = false] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testOuterInner[caching enabled = false] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testOuterInner[caching enabled = false] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testOuterOuter[caching enabled = false] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testOuterOuter[caching enabled = false] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testInnerWithRightVersionedOnly[caching enabled = false] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testInnerWithRightVersionedOnly[caching enabled = false] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testLeftWithLeftVersionedOnly[caching enabled = false] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testLeftWithLeftVersionedOnly[caching enabled = false] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testInnerWithLeftVersionedOnly[caching enabled = false] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testInnerWithLeftVersionedOnly[caching enabled = false] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
TaskAssignorIntegrationTest > shouldProperlyConfigureTheAssignor STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
TaskAssignorIntegrationTest > shouldProperlyConfigureTheAssignor PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
TaskMetadataIntegrationTest > shouldReportCorrectEndOffsetInformation STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
TaskMetadataIntegrationTest > shouldReportCorrectEndOffsetInformation PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
TaskMetadataIntegrationTest > shouldReportCorrectCommittedOffsetInformation 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
TaskMetadataIntegrationTest > shouldReportCorrectCommittedOffsetInformation 
PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
EmitOnChangeIntegrationTest > shouldEmitSameRecordAfterFailover() STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
EmitOnChangeIntegrationTest > shouldEmitSameRecordAfterFailover() PASSED

Gradle Test Run :streams:integrationTest > Gradle Test 

Re: [DISCUSS] KIP-910: Update Source offsets for Source Connectors without producing records

[Sagar]

Hi Yash/Chris,

Thanks for the feedback! I have updated the KIP with the suggestions
provided. I would also update the PR with the suggestions.

Also, I was hoping that this could make it to the 3.6 release given that it
would benefit source connectors which have some of the problems listed in
the Motivation Section.

Responses Inline:

Yash:

1) In the proposed changes section where you talk about modifying the
> offsets, could you please clarify that tasks shouldn't modify the offsets
> map that is passed as an argument? Currently, the distinction between the
> offsets map passed as an argument and the offsets map that is returned is
> not very clear in numerous places.



Added

2) The default return value of Optional.empty() seems to be fairly
> non-intuitive considering that the return value is supposed to be the
> offsets that are to be committed. Can we consider simply returning the
> offsets argument itself by default instead?



Chris is suggesting returning null for the default case. I am thinking to
make null
as the default return type. If the returned map is null, there won't be any
further
processing otherwise we will contonue with the existing logic.

3) The KIP states that "It is also possible that a task might choose to
> send a tombstone record as an offset. This is not recommended and to
> prevent connectors shooting themselves in the foot due to this" - could you
> please clarify why this is not recommended / supported?



I have added that a better way of doing that would be via KIP-875. Also, I
didn't want to include
any mechamisms for users to meddle with the offsets topic. Allowing
tombstone records via this method
would be akin to publishing tombstone records directly to the offsets topic
which is not recommended
generally.

4) The KIP states that "If a task returns an Optional of a null object or
> an Optional of an empty map, even for such cases the behaviour would would
> be disabled." - since this is an optional API that source task
> implementations don't necessarily need to implement, I don't think I fully
> follow why the return type of the proposed "updateOffsets" method is an
> Optional? Can we not simply use the Map as the return type instead?



Yeah, I updated the return type to be a Map.


5) The KIP states that "The offsets passed to the updateOffsets  method
> would be the offset from the latest source record amongst all source
> records per partition. This way, if the source offset for a given source
> partition is updated, that offset is the one that gets committed for the
> source partition." - we should clarify that the "latest" offset refers to
> the offsets that are about to be committed, and not the latest offsets
> returned from SourceTask::poll so far (see related discussion in
> https://issues.apache.org/jira/browse/KAFKA-15091 and
> https://issues.apache.org/jira/browse/KAFKA-5716).



Done


6) We haven't used the terminology of "Atleast Once Semantics" elsewhere in
> Connect since the framework itself does not (and cannot) make any
> guarantees on the delivery semantics. Depending on the source connector and
> the source system, both at-least once and at-most once semantics (for
> example - a source system where reads are destructive) are possible. We
> should avoid introducing this terminology in the KIP and instead refer to
> this scenario as exactly-once support being disabled.



Done


7) Similar to the above point, we should remove the use of the term
> "Exactly Once Semantics" and instead refer to exactly-once support being
> enabled since the framework can't guarantee exactly-once semantics for all
> possible source connectors (for example - a message queue source connector
> where offsets are essentially managed in the source system via an ack
> mechanism).


Done

8) In a previous attempt to fix this gap in functionality, a significant
> concern was raised on offsets ordering guarantees when we retry sending a
> batch of records (ref -
> https://github.com/apache/kafka/pull/5553/files#r213329307). It doesn't
> look like this KIP addresses that concern either? In the case where
> exactly-once support is disabled - if we update the committableOffsets with
> the offsets provided by the task through the new updateOffsets method,
> these offsets could be committed before older "regular" offsets are
> committed due to producer retries which could then lead to an inconsistency
> if the send operation eventually succeeds.




Thanks for bringing this up. I went through the comment shared above. If
you see the implementation
that I have in the PR, in EOS-disabled case, updateOffsets is invoked only
when toSend is null. Refer
here:
https://github.com/apache/kafka/pull/13899/files#diff-a3107b56382b6ec950dc9d19d21f188c21d4bf41853e0505d60d3bf87adab6a9R324-R330


Which means that we invoke updateOffsets only when
1) Either the last poll invocation didn't return any records or
2) All the records returned by the previous poll invocation got processed

Re: [ANNOUNCE] New committer: Greg Harris

[Jorge Esteban Quilcate Otoya]

Congrats Greg!!

On Tue 11. Jul 2023 at 15.20, Federico Valeri  wrote:

> Congrats Greg!
>
> On Tue, Jul 11, 2023 at 3:55 AM Luke Chen  wrote:
> >
> > Congrats Greg!
> >
> > Luke
> >
> > On Tue, Jul 11, 2023 at 8:19 AM Matthew de Detrich
> >  wrote:
> >
> > > Congratulations, well deserved!
> > >
> > > On Mon, Jul 10, 2023 at 5:45 PM Chris Egerton 
> wrote:
> > >
> > > > Hi all,
> > > >
> > > > The PMC for Apache Kafka has invited Greg Harris to become a
> committer,
> > > and
> > > > we are happy to announce that he has accepted!
> > > >
> > > > Greg has been contributing to Kafka since 2019. He has made over 50
> > > commits
> > > > mostly around Kafka Connect and Mirror Maker 2. His most notable
> > > > contributions include KIP-898: "Modernize Connect plugin discovery"
> and a
> > > > deep overhaul of the offset syncing logic in MM2 that addressed
> several
> > > > technically-difficult, long-standing, high-impact issues.
> > > >
> > > > He has also been an active participant in discussions and reviews on
> the
> > > > mailing lists and on GitHub.
> > > >
> > > > Thanks for all of your contributions, Greg. Congratulations!
> > > >
> > >
> > >
> > > --
> > >
> > > Matthew de Detrich
> > >
> > > *Aiven Deutschland GmbH*
> > >
> > > Immanuelkirchstraße 26, 10405 Berlin
> > >
> > > Amtsgericht Charlottenburg, HRB 209739 B
> > >
> > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > >
> > > *m:* +491603708037
> > >
> > > *w:* aiven.io *e:* matthew.dedetr...@aiven.io
> > >
>

Re: [DISCUSS] KIP-932: Queues for Kafka

[Andrew Schofield]

Hi Matthias,
Thanks for your comments. This KIP is clearly quite a big piece of work and 
it’s not complete at this stage. I think it’s a good
principle to develop it in the community. I’m prototyping the code and will 
soon start filling in some of the missing details
with concrete proposals for changes to the protocol and so on.

(1) The “fetch” request from a share-group consumer will be recognisable as 
such. The broker will know not to offer a preferred
replica for fetch-from-follower.

(2) Pattern subscription is not included in this initial KIP to reduce 
complexity. A pattern subscription has the characteristic
that the set of matching topics can change after the subscription has already 
been established.

(3) I think that “auto.offset.reset” should not be supported for share groups. 
Because each consumer gets its own configuration,
the initial offset is essentially unpredictable and I prefer to be explicit in 
this. If you want the share-group to start
at the latest offset, you need do nothing other than start your consumers. If 
you want the share-group to start at the earliest
offset, you need to use an Admin API or kafka-share-groups.sh. If you decide 
upon the earliest offset and use a topic which
has copious amounts of tiered data, there’s quite a performance implication to 
the decision. That’s why I have done it like this.

(3B) If the data is purged broker-side, the SPSO will leap forwards over the 
gap in a similar way as a consumer group
leaps over records which are purged.

(3C) Good question. The purged records logically become ARCHIVED.

(3D) Most messaging systems have the idea of message expiration. There’s an API 
option to say “this message expires after
X hours” and the messaging system will silently discard expired messages when 
their expiration intervals have elapsed.
Kafka does offer a kind of expiration in the form of retention, but it’s done 
in terms of the age of log segments rather than
the individual records within. There’s also the option of retention based on 
size.

In this environment, the question is how best to offer queuing semantics while 
working with the retention and log cleaning
behaviour that Kafka has. Personally, I think that time-based log retention 
offers an approximate equivalence of message
expiration in other systems. You typically do not know which unconsumed 
messages were expired. I can see that it would
perhaps be interesting to have metrics for this, but knowing precisely which 
records were archived without being acked
seems tricky.

Using a share-group with a compacted topic would give quite an unusual 
behaviour for a queue, but then it’s also an
unusual behaviour for a topic. I personally would avoid this combination.

(4) The design is per-message, but the code will reflect the fact that Kafka 
itself is more per-batch. Here’s an example of
what I mean.

When the first share-group consumer fetches records to process, the records 
will be fetched using the replica
manager. When reading from the log, it’s much cheaper to deal in batches of 
records rather than iterating over the
individual records. If a new batch of records is added to the in-flight records 
(so none of them has yet been delivered
for this share-partition), the batch of records will all be returned to one 
consumer. If the consumer processes and acknowledges
all of the records in the batch, this is more efficient than a mixture of 
acknowledgements and rejections.

In more complicated situations where perhaps one record in a batch was released 
but the rest were acknowledged,
we now have to re-deliver that individual record. A share-consumer’s fetch 
response will be able to contain a record which
is no longer being delivered with the rest of its original batch, but it’s less 
efficient.

This is what I mean by “prefer”. When it’s possible to deliver and acknowledge 
batches in their entirety, that’s what the
code will do and it will be at its most efficient. When batches become split, 
it will just be a bit more work.

I think this covers (4B), (4D) and (4E).

(4A) We keep record states per record. We could use any number of encoding 
formats to optimise the space this requires.

(4C) My current thinking is that each consumer will be allowed to express a 
preference for how much data can be returned
on each fetch request.

(5A) When I was writing the KIP, I played around with a few different ways of 
acknowledging the records and the API in the
KIP is the result of that thinking. Since most applications will simply iterate 
along the ConsumerRecords, saying that the
acknowledgements have to be in the same order doesn’t seem that burdensome.

(5B) There’s no shortcut way to release or reject all records in a batch at 
once. If you want to continue processing afterwards,
you must reject or release each record. If you want to give up processing, just 
close the consumer which releases the records.

(5C) ConsumerRecords could indeed include records from different partitions.

[jira] [Created] (KAFKA-15179) Add integration tests for the FileStream Sink and Source connectors

[Yash Mayya (Jira)]

Yash Mayya created KAFKA-15179:
--

 Summary: Add integration tests for the FileStream Sink and Source 
connectors
 Key: KAFKA-15179
 URL: https://issues.apache.org/jira/browse/KAFKA-15179
 Project: Kafka
  Issue Type: Improvement
Reporter: Yash Mayya
Assignee: Yash Mayya


Add integration tests for the FileStream Sink and Source connectors covering 
various different common scenarios.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15178) Poor performance of ConsumerCoordinator with many TopicPartitions

[Nicholas Telford (Jira)]

Nicholas Telford created KAFKA-15178:


 Summary: Poor performance of ConsumerCoordinator with many 
TopicPartitions
 Key: KAFKA-15178
 URL: https://issues.apache.org/jira/browse/KAFKA-15178
 Project: Kafka
  Issue Type: Bug
  Components: consumer
Affects Versions: 3.5.0
Reporter: Nicholas Telford
Assignee: Nicholas Telford
 Attachments: pollPhase.png

Doing some profiling of my Kafka Streams application, I noticed that the 
{{pollPhase}} suffers from a minor performance issue.

See flame graph below.  !pollPhase.png|width=1028,height=308!

{{ConsumerCoordinator.poll}} calls {{{}rejoinNeededOrPending{}}}, which checks 
the current {{metadataSnapshot}} against the {{{}assignmentSnapshot{}}}. This 
comparison is a deep-equality check, and if there's a large number of 
topic-partitions being consumed by the application, then this comparison can 
perform poorly.

I suspect this can be trivially addressed with a {{boolean}} flag that 
indicates when the {{metadataSnapshot}} has been updated (or is "dirty"), and 
actually needs to be checked, since most of the time it should be identical to 
{{{}assignmentSnapshot{}}}.

I plan to raise a PR with this optimization to address this issue.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.5 #33

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 561343 lines...]
> Task :metadata:testClasses UP-TO-DATE
> Task :connect:json:compileTestJava UP-TO-DATE
> Task :connect:json:testClasses UP-TO-DATE
> Task :connect:json:testJar
> Task :connect:json:testSrcJar
> Task :clients:generateMetadataFileForMavenJavaPublication

> Task :connect:api:javadoc
/home/jenkins/workspace/Kafka_kafka_3.5/connect/api/src/main/java/org/apache/kafka/connect/source/SourceRecord.java:44:
 warning - Tag @link: reference not found: org.apache.kafka.connect.data
1 warning

> Task :connect:api:copyDependantLibs UP-TO-DATE
> Task :connect:api:jar UP-TO-DATE
> Task :connect:api:generateMetadataFileForMavenJavaPublication
> Task :connect:json:copyDependantLibs UP-TO-DATE
> Task :connect:json:jar UP-TO-DATE
> Task :connect:json:generateMetadataFileForMavenJavaPublication
> Task :connect:api:javadocJar
> Task :connect:api:compileTestJava UP-TO-DATE
> Task :connect:api:testClasses UP-TO-DATE
> Task :connect:api:testJar
> Task :connect:json:publishMavenJavaPublicationToMavenLocal
> Task :connect:json:publishToMavenLocal
> Task :connect:api:testSrcJar
> Task :connect:api:publishMavenJavaPublicationToMavenLocal
> Task :connect:api:publishToMavenLocal
> Task :streams:javadoc
> Task :streams:copyDependantLibs UP-TO-DATE
> Task :streams:jar UP-TO-DATE
> Task :streams:generateMetadataFileForMavenJavaPublication
> Task :streams:javadocJar

> Task :clients:javadoc
/home/jenkins/workspace/Kafka_kafka_3.5/clients/src/main/java/org/apache/kafka/clients/admin/ScramMechanism.java:32:
 warning - Tag @see: missing final '>': "https://cwiki.apache.org/confluence/display/KAFKA/KIP-554%3A+Add+Broker-side+SCRAM+Config+API;>KIP-554:
 Add Broker-side SCRAM Config API

 This code is duplicated in 
org.apache.kafka.common.security.scram.internals.ScramMechanism.
 The type field in both files must match and must not change. The type field
 is used both for passing ScramCredentialUpsertion and for the internal
 UserScramCredentialRecord. Do not change the type field."
/home/jenkins/workspace/Kafka_kafka_3.5/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/secured/package-info.java:21:
 warning - Tag @link: reference not found: 
org.apache.kafka.common.security.oauthbearer
2 warnings

> Task :clients:javadocJar
> Task :clients:srcJar
> Task :clients:testJar
> Task :clients:testSrcJar
> Task :clients:publishMavenJavaPublicationToMavenLocal
> Task :clients:publishToMavenLocal
> Task :core:compileScala
> Task :core:classes
> Task :core:compileTestJava NO-SOURCE
> Task :core:compileTestScala
> Task :core:testClasses
> Task :streams:compileTestJava UP-TO-DATE
> Task :streams:testClasses UP-TO-DATE
> Task :streams:testJar
> Task :streams:testSrcJar
> Task :streams:publishMavenJavaPublicationToMavenLocal
> Task :streams:publishToMavenLocal

Deprecated Gradle features were used in this build, making it incompatible with 
Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings 
and determine if they come from your own scripts or plugins.

See 
https://docs.gradle.org/8.0.2/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 3m 10s
89 actionable tasks: 33 executed, 56 up-to-date
[Pipeline] sh
+ grep ^version= gradle.properties
+ cut -d= -f 2
[Pipeline] dir
Running in /home/jenkins/workspace/Kafka_kafka_3.5/streams/quickstart
[Pipeline] {
[Pipeline] sh
+ mvn clean install -Dgpg.skip
[INFO] Scanning for projects...
[INFO] 
[INFO] Reactor Build Order:
[INFO] 
[INFO] Kafka Streams :: Quickstart[pom]
[INFO] streams-quickstart-java[maven-archetype]
[INFO] 
[INFO] < org.apache.kafka:streams-quickstart >-
[INFO] Building Kafka Streams :: Quickstart 3.5.1-SNAPSHOT[1/2]
[INFO]   from pom.xml
[INFO] [ pom ]-
[INFO] 
[INFO] --- clean:3.0.0:clean (default-clean) @ streams-quickstart ---
[INFO] 
[INFO] --- remote-resources:1.5:process (process-resource-bundles) @ 
streams-quickstart ---
[INFO] 
[INFO] --- site:3.5.1:attach-descriptor (attach-descriptor) @ 
streams-quickstart ---
[INFO] 
[INFO] --- gpg:1.6:sign (sign-artifacts) @ streams-quickstart ---
[INFO] 
[INFO] --- install:2.5.2:install (default-install) @ streams-quickstart ---
[INFO] Installing 
/home/jenkins/workspace/Kafka_kafka_3.5/streams/quickstart/pom.xml to 
/home/jenkins/.m2/repository/org/apache/kafka/streams-quickstart/3.5.1-SNAPSHOT/streams-quickstart-3.5.1-SNAPSHOT.pom
[INFO] 
[INFO] --< org.apache.kafka:streams-quickstart-java >--
[INFO] Building streams-quickstart-java 3.5.1-SNAPSHOT[2/2]
[INFO]   from java/pom.xml
[INFO] 

Re: [ANNOUNCE] New committer: Greg Harris

[Federico Valeri]

Congrats Greg!

On Tue, Jul 11, 2023 at 3:55 AM Luke Chen  wrote:
>
> Congrats Greg!
>
> Luke
>
> On Tue, Jul 11, 2023 at 8:19 AM Matthew de Detrich
>  wrote:
>
> > Congratulations, well deserved!
> >
> > On Mon, Jul 10, 2023 at 5:45 PM Chris Egerton  wrote:
> >
> > > Hi all,
> > >
> > > The PMC for Apache Kafka has invited Greg Harris to become a committer,
> > and
> > > we are happy to announce that he has accepted!
> > >
> > > Greg has been contributing to Kafka since 2019. He has made over 50
> > commits
> > > mostly around Kafka Connect and Mirror Maker 2. His most notable
> > > contributions include KIP-898: "Modernize Connect plugin discovery" and a
> > > deep overhaul of the offset syncing logic in MM2 that addressed several
> > > technically-difficult, long-standing, high-impact issues.
> > >
> > > He has also been an active participant in discussions and reviews on the
> > > mailing lists and on GitHub.
> > >
> > > Thanks for all of your contributions, Greg. Congratulations!
> > >
> >
> >
> > --
> >
> > Matthew de Detrich
> >
> > *Aiven Deutschland GmbH*
> >
> > Immanuelkirchstraße 26, 10405 Berlin
> >
> > Amtsgericht Charlottenburg, HRB 209739 B
> >
> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> >
> > *m:* +491603708037
> >
> > *w:* aiven.io *e:* matthew.dedetr...@aiven.io
> >

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1987

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-932: Queues for Kafka

[Andrew Schofield]

Hi Kamal,
Thanks for your comments. Sorry for the delay. Just back from vacation.

101) When I was drafting the KIP, I did try an approach which allowed arbitrary 
order
of acknowledgement in the consumer but I wasn’t happy with the usability.

The records delivered to a specific consumer in a batch must be acknowledged in
the order delivered in that batch. But the overall order of records on the 
share-partition are
not guaranteed to be delivered in order because there might be several consumers
and retries and so on.

102) I think that it would be useful in some situations to be able to extend 
the lock timeout for
records in order to buy extra time to process them. I see that as a future 
extension to the Java
interface, rather than changing the configuration value that applies to all 
records acquired by
the consumer.

103) I’m prototyping the code and will be able to answer the configuration 
question properly
quite soon. I expect almost all of the existing consumer configs will apply to 
share-group consumers.

104) I think quotas will work the same for existing consumers and share-group 
consumers.

Hope this helps.
Andrew

> On 1 Jul 2023, at 12:42, Kamal Chandraprakash 
>  wrote:
>
> Hi Andrew,
>
> Thank you for the KIP -- interesting read. I have some questions:
>
> 101. "The calls to KafkaConsumer.acknowledge(ConsumerRecord,
> AcknowledgeType) must be
> issued in the order in which the records appear in the ConsumerRecords
> object, which will
> be in order of increasing offset for each share-partition"
>
> If the share-consumer uses thread pool internally and acknowledges the
> records in out-of-order fashion.
> Will this use case be supported? The "Managing durable share-partition
> state" have transitions where the
> records are ack'ed in out-of-order fashion so want to confirm this.
>
> 102. Will the configs be maintained in fine-grain per topic-to-share-group?
> Some share-consumer groups
> may want to increase the "record.lock.duration.ms" dynamically if record
> processing is taking longer time
> than usual during external system outage/downtime.
>
> 103. Can we also define whether all the consumer configs are eligible for
> share-consumer-group. (eg)
> `max.poll.interval.ms` default is 5 mins. Will this config have any effect
> on the share consumers?
>
> 104. How will the consumer quota work? Will it be similar to the existing
> consumer quota mechanism?
>
> --
> Kamal
>
> On Wed, Jun 7, 2023 at 9:17 PM Andrew Schofield <
> andrew_schofield_j...@outlook.com> wrote:
>
>> Hi Daniel,
>> True, I see your point. It’s analogous to a KafkaConsumer fetching
>> uncommitted records but not delivering them to the application.
>>
>> Thanks,
>> Andrew
>>
>>> On 7 Jun 2023, at 16:38, Dániel Urbán  wrote:
>>>
>>> Hi Andrew,
>>>
>>> I think the "pending" state could be the solution for reading beyond the
>>> LSO. Pending could indicate that a message is not yet available for
>>> consumption (so they won't be offered for consumers), but with
>> transactions
>>> ending, they can become "available". With a pending state, records
>> wouldn't
>>> "disappear", they would simply not show up until they become available on
>>> commit, or archived on abort.
>>>
>>> Regardless, I understand that this might be some extra, unwanted
>>> complexity, I just thought that with the message ordering guarantee gone,
>>> it would be a cool feature for share-groups. I've seen use-cases where
>> the
>>> LSO being blocked for an extended period of time caused huge lag for
>>> traditional read_committed consumers, which could be completely avoided
>> by
>>> share-groups.
>>>
>>> Thanks,
>>> Daniel
>>>
>>> Andrew Schofield  ezt írta (időpont:
>>> 2023. jún. 7., Sze, 17:28):
>>>
 Hi Daniel,
 Kind of. I don’t want a transaction abort to cause disappearance of
 records which are already in-flight. A “pending” state doesn’t seem
 helpful for read_committed. There’s no such disappearance problem
 for read_uncommitted.

 Thanks,
 Andrew

> On 7 Jun 2023, at 16:19, Dániel Urbán  wrote:
>
> Hi Andrew,
>
> I agree with having a single isolation.level for the whole group, it
 makes
> sense.
> As for:
> "b) The default isolation level for a share group is read_committed, in
> which case
> the SPSO and SPEO cannot move past the LSO."
>
> With this limitation (SPEO not moving beyond LSO), are you trying to
 avoid
> handling the complexity of some kind of a "pending" state for the
> uncommitted in-flight messages?
>
> Thanks,
> Daniel
>
> Andrew Schofield  ezt írta
>> (időpont:
> 2023. jún. 7., Sze, 16:52):
>
>> HI Daniel,
>> I’ve been thinking about this question and I think this area is a bit
>> tricky.
>>
>> If there are some consumers in a share group with isolation level
>> read_uncommitted
>> and other consumers with read_committed, they have different
 expectations
>> with

[jira] [Created] (KAFKA-15177) MirrorMaker 2 should implement the alterOffsets KIP-875 API

[Yash Mayya (Jira)]

Yash Mayya created KAFKA-15177:
--

 Summary: MirrorMaker 2 should implement the alterOffsets KIP-875 
API
 Key: KAFKA-15177
 URL: https://issues.apache.org/jira/browse/KAFKA-15177
 Project: Kafka
  Issue Type: Improvement
  Components: KafkaConnect, mirrormaker
Reporter: Yash Mayya


The {{MirrorSourceConnector}} class should implement the new alterOffsets API 
added in 
[KIP-875|https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect].
 We could also implement the API in 
{{MirrorCheckpointConnector}} and 
{{MirrorHeartbeatConnector}} to prevent external modification of offsets since 
the operation wouldn't really make sense in their case.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15176) Add missing tests for remote storage metrics

[Luke Chen (Jira)]

Luke Chen created KAFKA-15176:
-

 Summary: Add missing tests for remote storage metrics
 Key: KAFKA-15176
 URL: https://issues.apache.org/jira/browse/KAFKA-15176
 Project: Kafka
  Issue Type: Sub-task
Reporter: Luke Chen
Assignee: Luke Chen


{{RemoteLogReaderTaskQueueSize}}

{{RemoteLogReaderAvgIdlePercent}}

{{RemoteLogManagerTasksAvgIdlePercent}}

{{}}

https://github.com/apache/kafka/pull/13944#pullrequestreview-1513943273{{{}{}}}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15137) Don't log the entire request in KRaftControllerChannelManager

[Divij Vaidya (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Divij Vaidya resolved KAFKA-15137.
--
Resolution: Fixed

> Don't log the entire request in KRaftControllerChannelManager
> -
>
> Key: KAFKA-15137
> URL: https://issues.apache.org/jira/browse/KAFKA-15137
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 3.5.0, 3.6.0
>Reporter: David Arthur
>Assignee: Alyssa Huang
>Priority: Major
> Fix For: 3.6.0, 3.5.1
>
>
> While debugging some junit tests, I noticed some really long log lines in 
> KRaftControllerChannelManager. When the broker is down, we log a WARN that 
> includes the entire UpdateMetadataRequest or LeaderAndIsrRequest. For large 
> clusters, these can be really large requests, so this could potentially cause 
> excessive output in the log4j logs.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15155) Follow PEP 8 best practice in Python to check if a container is empty

[Divij Vaidya (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Divij Vaidya resolved KAFKA-15155.
--
Resolution: Fixed

> Follow PEP 8 best practice in Python to check if a container is empty
> -
>
> Key: KAFKA-15155
> URL: https://issues.apache.org/jira/browse/KAFKA-15155
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Divij Vaidya
>Assignee: Yi-Sheng Lien
>Priority: Trivial
>  Labels: newbie
> Fix For: 3.6.0
>
>
> *This is a good task for first time contributors to Kafka*
> At release.py Line:94 and at Line:60, we don't follow PEP 8 [1] best 
> practices.
> To check if a container or sequence (string, list, tuple) is empty, use if 
> not val. Do not compare its length using if len(val) == 0
> [1] 
> [https://peps.python.org/pep-0008/#programming-recommendations#:~:text=if%20not%20seq]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)