Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1958

[Apache Jenkins Server]

See 

Re: [DISCUSS] Apache Kafka 3.5.1 release

[Luke Chen]

Hi Divij,

Thanks for volunteering!

Luke

On Wed, Jun 28, 2023 at 11:54 PM Manyanda Chitimbo <
manyanda.chiti...@gmail.com> wrote:

> Thank you Divij for volunteering to perform the release.
>
> On Wed 28 Jun 2023 at 13:52, Divij Vaidya  wrote:
>
> > Hey folks
> >
> > Looks like we are ready to perform a release for 3.5.1 to provide a fix
> for
> > the vulnerability in snappy-java [1]
> >
> > I would like to volunteer as release manager for the 3.5.1 release.
> >
> > If there are no objections, I will start a release plan next Monday, on
> 3rd
> > July.
> >
> > [1] https://nvd.nist.gov/vuln/detail/CVE-2023-34455
> >
> > --
> > Divij Vaidya
> >
> --
> Manyanda Chitimbo.
>

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #1957

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-759: Unneeded repartition canceling

[Shay Lin]

Hi all,

Great discussion thread. May I take this KIP up? If it’s alright my plan is
to update the KIP with the operator `markAsPartitioned()`.

As you have discussed and pointed out, there are implications to downstream
joins or aggregation operations. Still, the operator is intended for
advanced users so my two cents is it would be a valuable addition
nonetheless. We could add this as a caution/consideration as part of the
java doc.

Let me know, thanks.
Shay

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1956

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 579505 lines...]
[INFO] --< org.apache.kafka:streams-quickstart-java >--
[INFO] Building streams-quickstart-java 3.6.0-SNAPSHOT[2/2]
[INFO]   from java/pom.xml
[INFO] --[ maven-archetype ]---
[INFO] 
[INFO] --- clean:3.0.0:clean (default-clean) @ streams-quickstart-java ---
[INFO] 
[INFO] --- remote-resources:1.5:process (process-resource-bundles) @ 
streams-quickstart-java ---
[INFO] 
[INFO] --- resources:2.7:resources (default-resources) @ 
streams-quickstart-java ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 6 resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- resources:2.7:testResources (default-testResources) @ 
streams-quickstart-java ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- archetype:2.2:jar (default-jar) @ streams-quickstart-java ---
[INFO] Building archetype jar: 
/home/jenkins/workspace/Kafka_kafka_trunk/streams/quickstart/java/target/streams-quickstart-java-3.6.0-SNAPSHOT
[INFO] 
[INFO] --- site:3.5.1:attach-descriptor (attach-descriptor) @ 
streams-quickstart-java ---
[INFO] 
[INFO] --- archetype:2.2:integration-test (default-integration-test) @ 
streams-quickstart-java ---
[INFO] 
[INFO] --- gpg:1.6:sign (sign-artifacts) @ streams-quickstart-java ---
[INFO] 
[INFO] --- install:2.5.2:install (default-install) @ streams-quickstart-java ---
[INFO] Installing 
/home/jenkins/workspace/Kafka_kafka_trunk/streams/quickstart/java/target/streams-quickstart-java-3.6.0-SNAPSHOT.jar
 to 
/home/jenkins/.m2/repository/org/apache/kafka/streams-quickstart-java/3.6.0-SNAPSHOT/streams-quickstart-java-3.6.0-SNAPSHOT.jar
[INFO] Installing 
/home/jenkins/workspace/Kafka_kafka_trunk/streams/quickstart/java/pom.xml to 
/home/jenkins/.m2/repository/org/apache/kafka/streams-quickstart-java/3.6.0-SNAPSHOT/streams-quickstart-java-3.6.0-SNAPSHOT.pom
[INFO] 
[INFO] --- archetype:2.2:update-local-catalog (default-update-local-catalog) @ 
streams-quickstart-java ---
[INFO] 
[INFO] Reactor Summary for Kafka Streams :: Quickstart 3.6.0-SNAPSHOT:
[INFO] 
[INFO] Kafka Streams :: Quickstart  SUCCESS [  2.535 s]
[INFO] streams-quickstart-java  SUCCESS [  1.603 s]
[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time:  4.537 s
[INFO] Finished at: 2023-06-28T21:44:38Z
[INFO] 
[WARNING] 
[WARNING] Plugin validation issues were detected in 7 plugin(s)
[WARNING] 
[WARNING]  * org.apache.maven.plugins:maven-remote-resources-plugin:1.5
[WARNING]  * org.apache.maven.plugins:maven-install-plugin:2.5.2
[WARNING]  * org.apache.maven.plugins:maven-archetype-plugin:2.2
[WARNING]  * org.apache.maven.plugins:maven-resources-plugin:2.7
[WARNING]  * org.apache.maven.plugins:maven-clean-plugin:3.0.0
[WARNING]  * org.apache.maven.plugins:maven-site-plugin:3.5.1
[WARNING]  * org.apache.maven.plugins:maven-gpg-plugin:1.6
[WARNING] 
[WARNING] For more or less details, use 'maven.plugin.validation' property with 
one of the values (case insensitive): [BRIEF, DEFAULT, VERBOSE]
[WARNING] 

Gradle Test Run :streams:integrationTest > Gradle Test Executor 179 > 
StoreQueryIntegrationTest > shouldQueryAllStalePartitionStores() PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 179 > 
StoreQueryIntegrationTest > 
shouldQuerySpecificStalePartitionStoresMultiStreamThreads() STARTED
[Pipeline] dir
Running in 
/home/jenkins/workspace/Kafka_kafka_trunk/streams/quickstart/test-streams-archetype
[Pipeline] {
[Pipeline] sh
+ echo Y
+ mvn archetype:generate -DarchetypeCatalog=local 
-DarchetypeGroupId=org.apache.kafka 
-DarchetypeArtifactId=streams-quickstart-java -DarchetypeVersion=3.6.0-SNAPSHOT 
-DgroupId=streams.examples -DartifactId=streams.examples -Dversion=0.1 
-Dpackage=myapps

Gradle Test Run :streams:integrationTest > Gradle Test Executor 179 > 
StoreQueryIntegrationTest > 
shouldQuerySpecificStalePartitionStoresMultiStreamThreads() PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 179 > 
StoreQueryIntegrationTest > shouldQuerySpecificStalePartitionStores() STARTED
[INFO] Scanning for projects...
[INFO] 
[INFO] --< org.apache.maven:standalone-pom >---
[INFO] Building Maven Stub Project (No POM) 1
[INFO] [ pom ]-
[INFO] 
[INFO] >>> archetype:3.2.1:generate (default-cli) > generate-sources @ 
standalone-pom >>>

[jira] [Resolved] (KAFKA-15078) When fetching offset 0 the KRaft leader should response with SnapshotId

[Jira]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

José Armando García Sancio resolved KAFKA-15078.

Fix Version/s: 3.6.0
   Resolution: Fixed

> When fetching offset 0 the KRaft leader should response with SnapshotId
> ---
>
> Key: KAFKA-15078
> URL: https://issues.apache.org/jira/browse/KAFKA-15078
> Project: Kafka
>  Issue Type: Improvement
>Reporter: José Armando García Sancio
>Assignee: José Armando García Sancio
>Priority: Major
> Fix For: 3.6.0
>
>
> With the current implementation if the follower fetches offset 0 and the 
> KRaft leader has a record batch at offset 0, it will always send a FETCH 
> response with records.
> If the KRaft log has generated a snapshot it is always more efficient of the 
> follower fetch the snapshot instead.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Final Reminder: Community Over Code call for presentations closing soon

[Rich Bowen]

[Note: You're receiving this email because you are subscribed to one or
more project dev@ mailing lists at the Apache Software Foundation.]

This is your final reminder that the Call for Presentations for
Community Over Code (formerly known as ApacheCon) is closing soon - on
Thursday, 13 July 2023 at 23:59:59 GMT.

https://communityovercode.org/call-for-presentations/

We are looking for talk proposals on all topics related to ASF projects
and open source software.

The event will be held in Halifax, Nova Scotia, Octiber 7th through
10th. More details about the event may be found on the event website at
https://communityovercode.org/

Rich, for the event planners

Re: [DISCUSS] KIP-933 Publish metrics when source connector fails to poll data

[Sagar]

Hey Ravindra,

Thanks for the KIP! It appears to be a useful addition to the metrics to
understand poll related failures which can go untracked as of now. I just
have a couple of minor comments:

1) Does it make sense to drop the *record *part from the metric name as it
doesn't seem to serve much purpose? I would rather call the metric as
*source-poll-errors-total
*and *source-poll-errors-rate*.
2) Staying on names, I am thinking, does it make more sense to have
*failures* in the name instead of *errors *i.e.*
source-poll-failures-total* and
*source-poll-failures-rate*? What do you think?
3) Regarding the inclusion of retriable exceptions, as of today, source
tasks don't retry even in cases of RetriableException. A PR was created to
modify this behaviour (https://github.com/apache/kafka/pull/13726) but the
reason I bring it up is that in that PR, the failures etc for retry context
would be computed from the RetryWithToleranceOperator. I am not sure when
would that get merged, but does it change the failure counting logic in any
ways?

Thanks!
Sagar.


On Sun, Jun 25, 2023 at 12:40 AM Ravindra Nath Kakarla <
ravindhran...@gmail.com> wrote:

> Hello,
>
> I would like to start a discussion on KIP-933 to add new metrics to Kafka
> Connect that helps  monitoring polling failures with source connectors.
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-933%3A+Publish+metrics+when+source+connector+fails+to+poll+data
>
> Looking forward to feedback on this.
>
> Thank you,
> Ravindranath
>

[jira] [Resolved] (KAFKA-15028) AddPartitionsToTxnManager metrics

[Justine Olshan (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Justine Olshan resolved KAFKA-15028.

Resolution: Fixed

> AddPartitionsToTxnManager metrics
> -
>
> Key: KAFKA-15028
> URL: https://issues.apache.org/jira/browse/KAFKA-15028
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Justine Olshan
>Assignee: Justine Olshan
>Priority: Major
> Attachments: latency-cpu.html
>
>
> KIP-890 added metrics for the AddPartitionsToTxnManager
> VerificationTimeMs – number of milliseconds from adding partition info to the 
> manager to the time the response is sent. This will include the round trip to 
> the transaction coordinator if it is called. This will also account for 
> verifications that fail before the coordinator is called.
> VerificationFailureRate – rate of verifications that returned in failure 
> either from the AddPartitionsToTxn response or through errors in the manager.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1955

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 580468 lines...]

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
VersionedKeyValueStoreIntegrationTest > shouldRestore STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
VersionedKeyValueStoreIntegrationTest > shouldRestore PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
VersionedKeyValueStoreIntegrationTest > shouldPutGetAndDelete STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
VersionedKeyValueStoreIntegrationTest > shouldPutGetAndDelete PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
VersionedKeyValueStoreIntegrationTest > 
shouldManualUpgradeFromNonVersionedTimestampedToVersioned STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
VersionedKeyValueStoreIntegrationTest > 
shouldManualUpgradeFromNonVersionedTimestampedToVersioned PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
HandlingSourceTopicDeletionIntegrationTest > 
shouldThrowErrorAfterSourceTopicDeleted STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
HandlingSourceTopicDeletionIntegrationTest > 
shouldThrowErrorAfterSourceTopicDeleted PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
StreamsAssignmentScaleTest > testHighAvailabilityTaskAssignorLargeNumConsumers 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
StreamsAssignmentScaleTest > testHighAvailabilityTaskAssignorLargeNumConsumers 
PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorLargePartitionCount STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorLargePartitionCount PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorManyThreadsPerClient STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorManyThreadsPerClient PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
StreamsAssignmentScaleTest > testStickyTaskAssignorManyStandbys STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
StreamsAssignmentScaleTest > testStickyTaskAssignorManyStandbys PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
StreamsAssignmentScaleTest > testStickyTaskAssignorManyThreadsPerClient STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
StreamsAssignmentScaleTest > testStickyTaskAssignorManyThreadsPerClient PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
StreamsAssignmentScaleTest > testFallbackPriorTaskAssignorManyThreadsPerClient 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
StreamsAssignmentScaleTest > testFallbackPriorTaskAssignorManyThreadsPerClient 
PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
StreamsAssignmentScaleTest > testFallbackPriorTaskAssignorLargePartitionCount 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
StreamsAssignmentScaleTest > testFallbackPriorTaskAssignorLargePartitionCount 
PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
StreamsAssignmentScaleTest > testStickyTaskAssignorLargePartitionCount STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
StreamsAssignmentScaleTest > testStickyTaskAssignorLargePartitionCount PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
StreamsAssignmentScaleTest > testFallbackPriorTaskAssignorManyStandbys STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
StreamsAssignmentScaleTest > testFallbackPriorTaskAssignorManyStandbys PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
StreamsAssignmentScaleTest > testHighAvailabilityTaskAssignorManyStandbys 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
StreamsAssignmentScaleTest > testHighAvailabilityTaskAssignorManyStandbys PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
StreamsAssignmentScaleTest > testFallbackPriorTaskAssignorLargeNumConsumers 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
StreamsAssignmentScaleTest > testFallbackPriorTaskAssignorLargeNumConsumers 
PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
StreamsAssignmentScaleTest > testStickyTaskAssignorLargeNumConsumers STARTED

Gradle Test Run 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.5 #26

[Apache Jenkins Server]

See 

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.4 #146

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 525161 lines...]
Gradle Test Run :core:integrationTest > Gradle Test Executor 170 > 
KafkaZkClientTest > testCreateTopLevelPaths() PASSED

Gradle Test Run :core:integrationTest > Gradle Test Executor 170 > 
KafkaZkClientTest > testGetAllTopicsInClusterDoesNotTriggerWatch() STARTED

Gradle Test Run :core:integrationTest > Gradle Test Executor 170 > 
KafkaZkClientTest > testGetAllTopicsInClusterDoesNotTriggerWatch() PASSED

Gradle Test Run :core:integrationTest > Gradle Test Executor 170 > 
KafkaZkClientTest > testIsrChangeNotificationGetters() STARTED

Gradle Test Run :core:integrationTest > Gradle Test Executor 170 > 
KafkaZkClientTest > testIsrChangeNotificationGetters() PASSED

Gradle Test Run :core:integrationTest > Gradle Test Executor 170 > 
KafkaZkClientTest > testLogDirEventNotificationsDeletion() STARTED

Gradle Test Run :core:integrationTest > Gradle Test Executor 170 > 
KafkaZkClientTest > testLogDirEventNotificationsDeletion() PASSED

Gradle Test Run :core:integrationTest > Gradle Test Executor 170 > 
KafkaZkClientTest > testGetLogConfigs() STARTED

Gradle Test Run :core:integrationTest > Gradle Test Executor 170 > 
KafkaZkClientTest > testGetLogConfigs() PASSED

Gradle Test Run :core:integrationTest > Gradle Test Executor 170 > 
KafkaZkClientTest > testBrokerSequenceIdMethods() STARTED

Gradle Test Run :core:integrationTest > Gradle Test Executor 170 > 
KafkaZkClientTest > testBrokerSequenceIdMethods() PASSED

Gradle Test Run :core:integrationTest > Gradle Test Executor 170 > 
KafkaZkClientTest > testAclMethods() STARTED

Gradle Test Run :core:integrationTest > Gradle Test Executor 170 > 
KafkaZkClientTest > testAclMethods() PASSED

Gradle Test Run :core:integrationTest > Gradle Test Executor 170 > 
KafkaZkClientTest > testCreateSequentialPersistentPath() STARTED

Gradle Test Run :core:integrationTest > Gradle Test Executor 170 > 
KafkaZkClientTest > testCreateSequentialPersistentPath() PASSED

Gradle Test Run :core:integrationTest > Gradle Test Executor 170 > 
KafkaZkClientTest > testConditionalUpdatePath() STARTED

Gradle Test Run :core:integrationTest > Gradle Test Executor 170 > 
KafkaZkClientTest > testConditionalUpdatePath() PASSED

Gradle Test Run :core:integrationTest > Gradle Test Executor 170 > 
KafkaZkClientTest > testGetAllTopicsInClusterTriggersWatch() STARTED

Gradle Test Run :core:integrationTest > Gradle Test Executor 170 > 
KafkaZkClientTest > testGetAllTopicsInClusterTriggersWatch() PASSED

Gradle Test Run :core:integrationTest > Gradle Test Executor 170 > 
KafkaZkClientTest > testDeleteTopicZNode() STARTED

Gradle Test Run :core:integrationTest > Gradle Test Executor 170 > 
KafkaZkClientTest > testDeleteTopicZNode() PASSED

Gradle Test Run :core:integrationTest > Gradle Test Executor 170 > 
KafkaZkClientTest > testDeletePath() STARTED

Gradle Test Run :core:integrationTest > Gradle Test Executor 170 > 
KafkaZkClientTest > testDeletePath() PASSED

Gradle Test Run :core:integrationTest > Gradle Test Executor 170 > 
KafkaZkClientTest > testGetBrokerMethods() STARTED

Gradle Test Run :core:integrationTest > Gradle Test Executor 170 > 
KafkaZkClientTest > testGetBrokerMethods() PASSED

Gradle Test Run :core:integrationTest > Gradle Test Executor 170 > 
KafkaZkClientTest > testJuteMaxBufffer() STARTED

Gradle Test Run :core:integrationTest > Gradle Test Executor 170 > 
KafkaZkClientTest > testJuteMaxBufffer() PASSED

Gradle Test Run :core:integrationTest > Gradle Test Executor 170 > 
KafkaZkClientTest > testCreateTokenChangeNotification() STARTED

Gradle Test Run :core:integrationTest > Gradle Test Executor 170 > 
KafkaZkClientTest > testCreateTokenChangeNotification() PASSED

Gradle Test Run :core:integrationTest > Gradle Test Executor 170 > 
KafkaZkClientTest > testGetTopicsAndPartitions() STARTED

Gradle Test Run :core:integrationTest > Gradle Test Executor 170 > 
KafkaZkClientTest > testGetTopicsAndPartitions() PASSED

Gradle Test Run :core:integrationTest > Gradle Test Executor 170 > 
KafkaZkClientTest > testChroot(boolean) > 
kafka.zk.KafkaZkClientTest.testChroot(boolean)[1] STARTED

Gradle Test Run :core:integrationTest > Gradle Test Executor 170 > 
KafkaZkClientTest > testChroot(boolean) > 
kafka.zk.KafkaZkClientTest.testChroot(boolean)[1] PASSED

Gradle Test Run :core:integrationTest > Gradle Test Executor 170 > 
KafkaZkClientTest > testChroot(boolean) > 
kafka.zk.KafkaZkClientTest.testChroot(boolean)[2] STARTED

Gradle Test Run :core:integrationTest > Gradle Test Executor 170 > 
KafkaZkClientTest > testChroot(boolean) > 
kafka.zk.KafkaZkClientTest.testChroot(boolean)[2] PASSED

Gradle Test Run :core:integrationTest > Gradle Test Executor 170 > 
KafkaZkClientTest > testRegisterBrokerInfo() STARTED

Gradle Test Run :core:integrationTest > Gradle Test Executor 170 > 
KafkaZkClientTest > testRegisterBrokerInfo() PASSED

[jira] [Created] (KAFKA-15133) RequestMetrics MessageConversionsTimeMs count is ticked even when no conversion occurs

[Edoardo Comar (Jira)]

Edoardo Comar created KAFKA-15133:
-

 Summary: RequestMetrics MessageConversionsTimeMs count is ticked 
even when no conversion occurs
 Key: KAFKA-15133
 URL: https://issues.apache.org/jira/browse/KAFKA-15133
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 3.4.1, 3.5.0
Reporter: Edoardo Comar
Assignee: Edoardo Comar


The Histogram 
{}{color:#00}RequestChannel{color}.{}}}messageConversionsTimeHist}}
is ticked even when a Produce/Fetch request incurred no conversion,
because a new entry is added to the historgram distribution, with a 0ms value.
 
It's confusing comparing the Histogram
kafka.network RequestMetrics MessageConversionsTimeMs
with the Meter
kafka.server BrokerTopicMetrics ProduceMessageConversionsPerSec
because for the latter, the metric is ticked only if a conversion actually 
occurred



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.3 #179

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 422017 lines...]
/home/jenkins/workspace/Kafka_kafka_3.3/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:155:
 warning - Tag @link: reference not found: this#isFailure()
/home/jenkins/workspace/Kafka_kafka_3.3/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:890:
 warning - Tag @link: reference not found: DefaultPartitioner
/home/jenkins/workspace/Kafka_kafka_3.3/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:919:
 warning - Tag @link: reference not found: DefaultPartitioner
/home/jenkins/workspace/Kafka_kafka_3.3/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:939:
 warning - Tag @link: reference not found: DefaultPartitioner
29 warnings

> Task :streams:javadocJar
> Task :streams:compileTestJava UP-TO-DATE
> Task :streams:testClasses UP-TO-DATE
> Task :streams:testJar
> Task :streams:testSrcJar

> Task :clients:javadoc
/home/jenkins/workspace/Kafka_kafka_3.3/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/secured/OAuthBearerLoginCallbackHandler.java:147:
 warning - Tag @link: reference not found: 

> Task :streams:publishMavenJavaPublicationToMavenLocal
> Task :streams:publishToMavenLocal

> Task :clients:javadoc
1 warning

> Task :clients:javadocJar
> Task :clients:testJar
> Task :clients:testSrcJar
> Task :clients:publishMavenJavaPublicationToMavenLocal
> Task :clients:publishToMavenLocal

Deprecated Gradle features were used in this build, making it incompatible with 
Gradle 8.0.

You can use '--warning-mode all' to show the individual deprecation warnings 
and determine if they come from your own scripts or plugins.

See 
https://docs.gradle.org/7.4.2/userguide/command_line_interface.html#sec:command_line_warnings

Execution optimizations have been disabled for 2 invalid unit(s) of work during 
this build to ensure correctness.
Please consult deprecation warnings for more details.

BUILD SUCCESSFUL in 45s
79 actionable tasks: 33 executed, 46 up-to-date
[Pipeline] sh
+ grep ^version= gradle.properties
+ cut -d= -f 2
[Pipeline] dir
Running in /home/jenkins/workspace/Kafka_kafka_3.3/streams/quickstart
[Pipeline] {
[Pipeline] sh
+ mvn clean install -Dgpg.skip
[INFO] Scanning for projects...
[INFO] 
[INFO] Reactor Build Order:
[INFO] 
[INFO] Kafka Streams :: Quickstart[pom]
[INFO] streams-quickstart-java[maven-archetype]
[INFO] 
[INFO] < org.apache.kafka:streams-quickstart >-
[INFO] Building Kafka Streams :: Quickstart 3.3.3-SNAPSHOT[1/2]
[INFO]   from pom.xml
[INFO] [ pom ]-
[INFO] 
[INFO] --- clean:3.0.0:clean (default-clean) @ streams-quickstart ---
[INFO] 
[INFO] --- remote-resources:1.5:process (process-resource-bundles) @ 
streams-quickstart ---
[INFO] 
[INFO] --- site:3.5.1:attach-descriptor (attach-descriptor) @ 
streams-quickstart ---
[INFO] 
[INFO] --- gpg:1.6:sign (sign-artifacts) @ streams-quickstart ---
[INFO] 
[INFO] --- install:2.5.2:install (default-install) @ streams-quickstart ---
[INFO] Installing 
/home/jenkins/workspace/Kafka_kafka_3.3/streams/quickstart/pom.xml to 
/home/jenkins/.m2/repository/org/apache/kafka/streams-quickstart/3.3.3-SNAPSHOT/streams-quickstart-3.3.3-SNAPSHOT.pom
[INFO] 
[INFO] --< org.apache.kafka:streams-quickstart-java >--
[INFO] Building streams-quickstart-java 3.3.3-SNAPSHOT[2/2]
[INFO]   from java/pom.xml
[INFO] --[ maven-archetype ]---
[INFO] 
[INFO] --- clean:3.0.0:clean (default-clean) @ streams-quickstart-java ---
[INFO] 
[INFO] --- remote-resources:1.5:process (process-resource-bundles) @ 
streams-quickstart-java ---
[INFO] 
[INFO] --- resources:2.7:resources (default-resources) @ 
streams-quickstart-java ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 6 resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- resources:2.7:testResources (default-testResources) @ 
streams-quickstart-java ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- archetype:2.2:jar (default-jar) @ streams-quickstart-java ---
[INFO] Building archetype jar: 
/home/jenkins/workspace/Kafka_kafka_3.3/streams/quickstart/java/target/streams-quickstart-java-3.3.3-SNAPSHOT
[INFO] 
[INFO] --- site:3.5.1:attach-descriptor (attach-descriptor) @ 
streams-quickstart-java ---
[INFO] 
[INFO] --- archetype:2.2:integration-test (default-integration-test) @ 
streams-quickstart-java ---
[INFO] 
[INFO] --- gpg:1.6:sign (sign-artifacts) @ streams-quickstart-java ---
[INFO] 
[INFO] --- 

[jira] [Created] (KAFKA-15132) Implement disable & re-enablement for Tiered Storage

[Divij Vaidya (Jira)]

Divij Vaidya created KAFKA-15132:


 Summary: Implement disable & re-enablement for Tiered Storage
 Key: KAFKA-15132
 URL: https://issues.apache.org/jira/browse/KAFKA-15132
 Project: Kafka
  Issue Type: New Feature
  Components: core
Reporter: Divij Vaidya
Assignee: Divij Vaidya


KIP-405 [1] introduces the Tiered Storage feature in Apache Kafka. One of the 
limitations mentioned in the KIP is inability to re-enable TS on a topic after 
it has been disabled.


{quote}Once tier storage is enabled for a topic, it can not be disabled. We 
will add this feature in future versions. One possible workaround is to create 
a new topic and copy the data from the desired offset and delete the old topic. 
{quote}

This task will propose a new KIP which extends on KIP-405 to describe the 
behaviour on on disablement and re-enablement of tiering storage for a topic. 
The solution will apply for both Zk and KRaft variants.


[1] KIP-405 - 
[https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage]
 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Apache Kafka 3.5.1 release

[Manyanda Chitimbo]

Thank you Divij for volunteering to perform the release.

On Wed 28 Jun 2023 at 13:52, Divij Vaidya  wrote:

> Hey folks
>
> Looks like we are ready to perform a release for 3.5.1 to provide a fix for
> the vulnerability in snappy-java [1]
>
> I would like to volunteer as release manager for the 3.5.1 release.
>
> If there are no objections, I will start a release plan next Monday, on 3rd
> July.
>
> [1] https://nvd.nist.gov/vuln/detail/CVE-2023-34455
>
> --
> Divij Vaidya
>
-- 
Manyanda Chitimbo.

Re: [DISCUSS] KIP-916: MM2 distributed mode flow log context

[Dániel Urbán]

If there are no further comments, I will kick off a vote soon for the KIP.

Dániel Urbán  ezt írta (időpont: 2023. jún. 12., H,
11:27):

> Updated the KIP with a few example log lines before/after the change.
> Daniel
>
> Dániel Urbán  ezt írta (időpont: 2023. jún. 7.,
> Sze, 13:59):
>
>> Hi Chris,
>>
>> Thank you for your comments! I updated the KIP. I still need to add the
>> example before/after log lines, will do that soon, but I addressed all the
>> other points.
>> 1. Added more details on thread renaming under Public Interfaces, removed
>> pseudo code.
>> 2. Removed the stale header - originally, client.id related changes were
>> part of the KIP, and I failed removing all leftovers of that version.
>> 3. Threads listed under Public Interfaces with current/proposed names.
>> 4. Added a comment in the connect-log4j.properties, similar to the one
>> introduced in KIP-449. We don't have a dedicated MM2 log4j config, not sure
>> if we should introduce it here.
>> 5. Clarified testing section - I think thread names should not be tested
>> (they never were), but testing will focus on the new MDC context value.
>>
>> Thanks,
>> Daniel
>>
>> Chris Egerton  ezt írta (időpont: 2023. jún.
>> 5., H, 16:46):
>>
>>> Hi Daniel,
>>>
>>> Thanks for the updates! A few more thoughts:
>>>
>>> 1. The "Proposed Changes" section seems a bit like a pseudocode
>>> implementation of the KIP. We don't really need this level of detail;
>>> most
>>> of the time, we're just looking for implementation details that don't
>>> directly affect the user-facing changes proposed in the "Public
>>> Interfaces"
>>> section but are worth mentioning because they (1) demonstrate how the
>>> user-facing changes are made possible, (2) indirectly affect user-facing
>>> behavior, or (3) go into more detail (like providing examples) about the
>>> user-facing behavior. For this KIP, I think examples of user-facing
>>> behavior (like before/after of thread names and log messages) and
>>> possibly
>>> an official description of the scope of the changes (which threads are
>>> going to be renamed and/or include the new MDC key, and which aren't?)
>>> are
>>> all that we'd really need in this section; everything else is fairly
>>> clear
>>> IMO. FWIW, the reason we want to discourage going into too much detail
>>> with
>>> KIPs is that it can quickly devolve into code review over mailing list,
>>> which can hold KIPs up for longer than necessary when the core design
>>> changes they contain are already basically accepted by everyone.
>>>
>>> 2. The "MM2 distributed mode client.id and log change" header seems
>>> like it
>>> may no longer be accurate; the contents do not mention any changes to
>>> client IDs. I might be missing something though; please correct me if I
>>> am.
>>>
>>> 3. Can you provide some before/after examples of what thread names and
>>> log
>>> messages will look like? I'm wondering about the thread that the
>>> distributed herder runs on, threads for connectors and tasks, and threads
>>> for polling internal topics (which we currently manage with the
>>> KafkaBasedLog class). It's fine if some of these are unchanged, I just
>>> want
>>> to better understand the scope of the proposed changes and get an idea of
>>> how they may appear to users.
>>>
>>> 4. There's no mention of changes to the default Log4j config files that
>>> we
>>> ship. Is this intentional? I feel like we need some way for users to
>>> easily
>>> discover this feature; if we're not going to touch our default Log4j
>>> config
>>> files, is there another way that we can expect users to find out about
>>> the
>>> new MDC key?
>>>
>>> 5. RE the "Test Plan" section: can you provide a little more detail of
>>> which cases we'll be covering with the proposed unit tests? Will we be
>>> verifying that the MDC context is set in various places? If so, which
>>> places? And the same with thread names? (There doesn't have to be a ton
>>> of
>>> detail, but a little more than "unit tests" would be nice )
>>>
>>> Cheers,
>>>
>>> Chris
>>>
>>> On Mon, Jun 5, 2023 at 9:45 AM Dániel Urbán 
>>> wrote:
>>>
>>> > I updated the KIP accordingly. Tried to come up with more generic
>>> terms in
>>> > the Connect code instead of referring to "flow" anywhere.
>>> > Daniel
>>> >
>>> > Dániel Urbán  ezt írta (időpont: 2023. jún.
>>> 5., H,
>>> > 14:49):
>>> >
>>> > > Hi Chris,
>>> > >
>>> > > Thank you for your comments!
>>> > >
>>> > > I agree that the toString based logging is not ideal, and I believe
>>> all
>>> > > occurrences are within a proper logging context, so they can be
>>> ignored.
>>> > > If thread names can be changed unconditionally, I agree, using a new
>>> MDC
>>> > > key is the ideal solution.
>>> > >
>>> > > Will update the KIP accordingly.
>>> > >
>>> > > Thanks,
>>> > > Daniel
>>> > >
>>> > > Chris Egerton  ezt írta (időpont: 2023.
>>> jún.
>>> > 2.,
>>> > > P, 22:23):
>>> > >
>>> > >> Hi Daniel,
>>> > >>
>>> > >> Are there any cases of Object::toString being used by 

Re: [DISCUSS] KIP-918: MM2 Topic And Group Listener

[Dániel Urbán]

Hi Chris,

Thank you for your comments, and sorry for the late reply.

1. Having a single, connector-reported metric for the topics, and another
one for the groups sounds good to me. The only issue I see here is that I'm
not familiar with any non-primitive metrics in the Kafka codebase, and
don't know if introducing a map/list type metric value will be a problem.
2. The intention is to provide the full set each time. A delta based
approach could be possible, but I think it would be an unnecessary
complication. If we go with a metric instead, we should just stick to the
full set.

I will update the KIP with the metric based approach.

Thanks,
Daniel

Chris Egerton  ezt írta (időpont: 2023. jún. 6.,
K, 16:32):

> Hi Daniel,
>
> Thanks for the KIP! I see the value in exposing information on replicated
> topics and groups. For one, it matches a similar feature added to Kafka
> Connect in KIP-558 [1], where we started tracking the set of topics that
> connectors interacted with over their lifetime. And there's also the use
> case you provided about identifying the provenance of topics replicated
> with the identity replication policy (or really, any policy that doesn't
> preserve information about the source cluster). Finally, it seems like a
> decent debugging aid for prototyping and initially standing up MM2
> clusters, and a liveness check for existing ones.
>
> Here are my thoughts so far:
>
> 1. I know that MM2 has a lot of pluggable interfaces already but I'm always
> a little hesitant to introduce another one. One alternative could be to add
> new metrics for the sets of replicated topics and groups. Users can already
> implement pluggable metrics reporters [2], which could be a substitute for
> the listener interface proposed in the KIP.
>
> 2. Is the intention to provide the listener with the total current set of
> replicated groups and topics every time that set is computed? Or is the
> listener given the complete set the first time and a delta other times?
> Based on the type signatures of the interface methods I'm guessing it's the
> former, but based on the names (which use the word "changed") it seems like
> the latter. If we use complete sets, I think "refreshed" or "computed" may
> be better as suffixes, or we could possibly use "replicated" as a prefix
> ("replicatedTopics", "replicatedGroups").
>
> [1] -
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-558%3A+Track+the+set+of+actively+used+topics+by+connectors+in+Kafka+Connect
> [2] -
>
> https://kafka.apache.org/34/javadoc/org/apache/kafka/common/metrics/MetricsReporter.html
>
> Cheers,
>
> Chris
>
> On Fri, Apr 21, 2023 at 9:00 AM Dániel Urbán 
> wrote:
>
> > Thanks for the comments Viktor.
> > 1. My original motivation was IdentityReplicationPolicy based monitoring.
> > The current MirrorClient implementation cannot list the replica topics of
> > the target cluster. I think relying on the topic-partition level metrics
> is
> > a complex solution. Instead, I would like to make it simple to collect
> all
> > the replicated topics of a flow, without relying on the name of the
> topics.
> > Then, I simply tried to generalize the approach.
> > 2. Checkpoint metrics are reported per (group, topic, partition), it
> means
> > that there is no metric associated with a group. If a filter picks up a
> > group, but the group doesn't have committed offsets for any of the
> > replicated partitions, there is no metric to be eagerly registered. This
> is
> > a difference between how topic replication and group checkpointing works
> -
> > empty topics are still picked up for partition creation and to consume
> from
> > them. Groups are only picked up if they have committed offsets already.
> > 3. Not exactly sure what is the added value of listing all
> > topic-partitions, but that information is available where the filtering
> > happens. For groups, we don't have anything else besides the group name,
> so
> > we cannot really provide more info at that point without significantly
> > changing the refresh group logic.
> >
> > Thanks,
> > Daniel
> >
> > Viktor Somogyi-Vass  ezt írta
> > (időpont: 2023. ápr. 21., P, 11:43):
> >
> > > Hi all,
> > >
> > > A couple of comments:
> > > 1) Regarding the motivation: is the motivation simply monitoring
> related
> > or
> > > are there any other reasons to this?
> > > 2) Can we change monitoring to be identical to filters, so that what is
> > > actively filtered, we monitor exactly those topics and groups? (So
> group
> > > metrics aren't added lazily when a checkpoint is created but when the
> > > filter is changed.)
> > > 3) Not sure if we want to widen the scope but since these are
> interfaces
> > > I'd use TopicPartition and some kind of GroupDescription classes (not
> > sure
> > > if the latter exists) instead of Strings. If later on we'll need extra
> > > properties for these then it can be added on easier.
> > >
> > > Best,
> > > Viktor
> > >
> > > On Wed, Apr 19, 2023 at 1:42 PM Dániel Urbán 
> > > 

Re: [DISCUSS] Apache Kafka 3.5.1 release

[Chris Egerton]

Thanks for volunteering, Divij!

On Wed, Jun 28, 2023 at 7:52 AM Divij Vaidya 
wrote:

> Hey folks
>
> Looks like we are ready to perform a release for 3.5.1 to provide a fix for
> the vulnerability in snappy-java [1]
>
> I would like to volunteer as release manager for the 3.5.1 release.
>
> If there are no objections, I will start a release plan next Monday, on 3rd
> July.
>
> [1] https://nvd.nist.gov/vuln/detail/CVE-2023-34455
>
> --
> Divij Vaidya
>

Re: Offsets: consumption and production in rollback

[Andrew Schofield]

Hi Henry,
Consumers get to choose an isolation level. There’s one instance I can think of 
where AdminClient also has
some ability to let users choose how to deal with uncommitted data. If you’ve 
not seen KIP-851 take a look:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-851%3A+Add+requireStable+flag+into+ListConsumerGroupOffsetsOptions
By your question, I expect you have seen it.

Control records are kind of invisibly lurking like dark matter. The approach 
I’ve taken with this kind of thing
is to cope with the fact that the offsets of the real records are increasing 
but not necessarily consecutive.
If I’m using READ_COMMITTED isolation level, there are also gaps when 
transactions roll back.
I design my consumers so they are not surprised by the gaps and they don’t try 
to calculate the number
of records. Instead, they just continually consume.

Hope this helps,
Andrew

> On 28 Jun 2023, at 09:28, Henry GALVEZ  wrote:
> 
> Hi Andrew,
> 
> Thank you for your response.
> 
> I understand your explanation, but in both cases, I am using an "isolation 
> level" of READ_COMMITTED. I believe the issue lies in the 
> AdminClient.listOffsets method, as it may not be considering the isolation 
> level, where as the consumer of AdminClient.listConsumerGroupOffsets does 
> consider it.
> 
> What are your thoughts on this?
> 
> Additionally, would it be more suitable to implement a solution that reverts 
> the offsets in case of transaction rollbacks? It's possible that there's a 
> logic aspect I'm not fully grasping.
> 
> Perhaps I need to utilize the internal control records and their offsets. 
> Could you point me in the right direction for their documentation?
> 
> Thank you,
> Henry
> 
> 
> De: Andrew Schofield 
> Enviado: martes, 27 de junio de 2023 13:22
> Para: dev@kafka.apache.org 
> Asunto: Fwd: Offsets: consumption and production in rollback
> 
> Hi Henry,
> Thanks for your message.
> 
> Kafka transactions are a bit unusual. If you produce a message inside a 
> transaction, it is assigned an offset on a topic-partition before
> the transaction even commits. That offset is not “revoked” if the transaction 
> rolls back.
> 
> This is why the consumer has the concept of “isolation level”. It essentially 
> controls whether the consumer can “see” the
> uncommitted or even rolled back messages.
> 
> A consumer using the committed isolation level only consumes committed 
> messages, but the offsets that it observes do
> reflect the uncommitted messages. So, if you observe the progress of the 
> offsets of the records consumed, you see that they
> skip the messages that were produced but then rolled back. There are also 
> invisible control records that are used to achieve
> transactional behaviour, and those also have offsets.
> 
> I’m not sure that this is really “bogus lag” but, when you’re using 
> transactions, there’s not a one-to-one relationship
> between offsets and consumable records.
> 
> Hope this helps,
> Andrew
> 
> Begin forwarded message:
> 
> From: Henry GALVEZ 
> Subject: Offsets: consumption and production in rollback
> Date: 27 June 2023 at 10:48:31 BST
> To: "us...@kafka.apache.org" , "dev@kafka.apache.org" 
> 
> Reply-To: dev@kafka.apache.org
> 
> I have some doubts regarding message consumption and production, as well as 
> transactional capabilities. I am using a Kafka template to produce a message 
> within a transaction. After that, I execute another transaction that produces 
> a message and intentionally throws a runtime exception to simulate a 
> transaction rollback.
> 
> Next, I use the Kafka AdminClient to retrieve the latest offset for the topic 
> partition and the consumer group's offsets for the same topic partition. 
> However, when I compare the offset numbers, I notice a difference. In this 
> example, the consumer has 4 offsets, while the topic has only 2.
> 
> I have come across references to this issue in a Spring-Kafka report, 
> specifically in the Kafka-10683 report, where developers describe it as 
> either Bogus or Pseudo Lag.
> 
> I am keen on resolving this problem, and I would greatly appreciate hearing 
> about your experiences and knowledge regarding this matter.
> 
> Thank you very much
> Henry
> 

[DISCUSS] Apache Kafka 3.5.1 release

[Divij Vaidya]

Hey folks

Looks like we are ready to perform a release for 3.5.1 to provide a fix for
the vulnerability in snappy-java [1]

I would like to volunteer as release manager for the 3.5.1 release.

If there are no objections, I will start a release plan next Monday, on 3rd
July.

[1] https://nvd.nist.gov/vuln/detail/CVE-2023-34455

--
Divij Vaidya

Re: [DISCUSS] KIP-940: Broker extension point for validating record contents at produce time

[Edoardo Comar]

Hi Andrew,

thanks for your comments ! Please see replies inline below.

On Mon, 26 Jun 2023 at 16:51, Andrew Schofield
 wrote:
> 4) For a new interface, I wonder whether it would be better to use 
> TopicIdPartition rather
> than TopicPartition. Topic IDs are gradually spreading across the public 
> interfaces for Kafka.

Thanks for the suggestion, we’ve updated the KIP.

> 5) The new topic config is called `record.validation.policy`. The javadoc for 
> the validationPolicy()
> method says `validation.policy`.

oops, fixed in the KIP, thx.

> 6) I’m surprised that you need a `HeaderProxy` interface when `Headers` and 
> `Header` are
> already interfaces. I would have expected it was possible to create proxy 
> instances of the
> headers using the existing interfaces with a little cunning.

A header value() returns a byte[] which can naturally be modified.
We introduced the HeaderProxy interface to make it clear that the
returned values are read-only,
without being forced to make deep copies of byte[].

Thanks,
Edo & Adrian

Re: [DISCUSS] KIP-940: Broker extension point for validating record contents at produce time

[Edoardo Comar]

Hi Tom,

thanks for tour comments, replies inline below.

On Thu, 22 Jun 2023 at 10:58, Tom Bentley  wrote:
>
> Hi Edorado and Adrian,
>
> Thanks for the KIP.
>
> I think it would be good to elaborate on exactly how validate() gets
> called, because I think there are a number of potential problems, or at
> least things to consider.
>
> From the broker's point of view, validate() can do arbitrary things. It
> might never return due to blocking or an infinite loop. It might cause an
> OOME, or throw a StackOverflowException. These are not entirely unlikely
> and the risks cannot always be easily avoided by a careful policy
> implementation. For example, a plugin which performs schema validation
> would typically be fetching schema information from a remote registry (and
> caching it for future use), and so could block on the networking (there are
> further questions about retry in the case of network error). Then, when
> deserializing a format like JSON deserializers might be prone to SOE or
> OOME (e.g. consider a naive recursive JSON parser with JSON input starting
> "..."). More generally, incorrect
> deserialization of untrusted input is a common kind of CVE. Similarly
> validation might involve regular expression matching (for example
> JSONSchema supports pattern constraints). The matcher implementation really
> matters and common matchers, including Java's Pattern, expose you to the
> possibility of nasty exponential time behaviour.

We agree with your observations, running 3rd party code inside the
broker exposes it to these problems.
The Authorizer for example, although it’s not typically involved with
user input deserialization and it is not invoked in a lock,
is an example of existing plugin code invoked from the IO threads and
implementations might access external systems.
Server side input validation carries a tradeoff between functionality
and risk, if it is not acceptable in a certain deployment then it
should not be enabled.

An implementation could use an own thread-pool and have the call
coming from the IO thread bounded by a timeout.
We do not think such a solution should be mandated as part of the
plugin interface.
We envision that the record validation plugin implementations used in
a production system to be production quality code,
likely developed and tested by the schema registry provider as are the serdes.
In fact there is a natural semantic coupling between the serdes and
the validator.
We do not expect Kafka cluster administrators to just run any code
within their brokers.

Furthermore, not all validation requires parsing of the message
payload to provide value.
For example, a policy that checks records carry a valid schema ID
would prevent common misconfigurations - like running a client without
a registry’s serdes.

> You mentioned LogValidator in the KIP. This executes on an IO thread and
> gets called with the log lock held. So the consequences of the validation
> blocking could actually be a real problem from a broker availability PoV if
> this validation happens in the same place. In the worst case all the IO
> threads get stuck because of bad input (perhaps from a single producer), or
> network problems between the broker and the registry. I don't think simply
> moving the validation to before the acquisition of the lock is an easy
> solution either, because of the dependency on the compression validation.

The existing LogValidator seems a very natural point to perform an
optional deeper validation than the existing one,
Again an implementation that uses a timeout-bounded call seems a possibility.

Thanks to your observation we think some metrics should be introduced
to monitor the plugin behaviour.
We could enhance the KIP introducing metrics similar to the existing
ones related to message conversions and invalid messages, e.g.

kafka.network:type=RequestMetrics,name=MessageValidationTimeMs

kafka.server:type=BrokerTopicMetrics,name=ProduceMessageValidationsPerSec
kafka.server:type=BrokerTopicMetrics,name=ProduceMessageValidationsPerSec,topic=

kafka.server:type=BrokerTopicMetrics,name=InvalidMessageRecordsPerSec
kafka.server:type=BrokerTopicMetrics,name=InvalidMessageRecordsPerSec,topic=

What do you think?

Thanks,
Edo & Adrian

[jira] [Created] (KAFKA-15131) Improve RemoteStorageManager exception handling

[Jorge Esteban Quilcate Otoya (Jira)]

Jorge Esteban Quilcate Otoya created KAFKA-15131:


 Summary: Improve RemoteStorageManager exception handling
 Key: KAFKA-15131
 URL: https://issues.apache.org/jira/browse/KAFKA-15131
 Project: Kafka
  Issue Type: Improvement
  Components: core
Reporter: Jorge Esteban Quilcate Otoya
Assignee: Jorge Esteban Quilcate Otoya


As discussed here[1], RemoteStorageManager javadocs requires clarification 
regarding error handling:
 * Remove ambiguity on `RemoteResourceNotFoundException` description
 * Describe when `RemoteResourceNotFoundException` can/should be thrown
 * Describe expectations for idempotent operations when copying/deleting

 

[1] 
https://issues.apache.org/jira/browse/KAFKA-7739?focusedCommentId=17720936=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17720936



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

RE: Offsets: consumption and production in rollback

[Henry GALVEZ]

Hi Andrew,

Thank you for your response.

I understand your explanation, but in both cases, I am using an "isolation 
level" of READ_COMMITTED. I believe the issue lies in the 
AdminClient.listOffsets method, as it may not be considering the isolation 
level, where as the consumer of AdminClient.listConsumerGroupOffsets does 
consider it.

What are your thoughts on this?

Additionally, would it be more suitable to implement a solution that reverts 
the offsets in case of transaction rollbacks? It's possible that there's a 
logic aspect I'm not fully grasping.

Perhaps I need to utilize the internal control records and their offsets. Could 
you point me in the right direction for their documentation?

Thank you,
Henry


De: Andrew Schofield 
Enviado: martes, 27 de junio de 2023 13:22
Para: dev@kafka.apache.org 
Asunto: Fwd: Offsets: consumption and production in rollback

Hi Henry,
Thanks for your message.

Kafka transactions are a bit unusual. If you produce a message inside a 
transaction, it is assigned an offset on a topic-partition before
the transaction even commits. That offset is not “revoked” if the transaction 
rolls back.

This is why the consumer has the concept of “isolation level”. It essentially 
controls whether the consumer can “see” the
uncommitted or even rolled back messages.

A consumer using the committed isolation level only consumes committed 
messages, but the offsets that it observes do
reflect the uncommitted messages. So, if you observe the progress of the 
offsets of the records consumed, you see that they
skip the messages that were produced but then rolled back. There are also 
invisible control records that are used to achieve
transactional behaviour, and those also have offsets.

I’m not sure that this is really “bogus lag” but, when you’re using 
transactions, there’s not a one-to-one relationship
between offsets and consumable records.

Hope this helps,
Andrew

Begin forwarded message:

From: Henry GALVEZ 
Subject: Offsets: consumption and production in rollback
Date: 27 June 2023 at 10:48:31 BST
To: "us...@kafka.apache.org" , "dev@kafka.apache.org" 

Reply-To: dev@kafka.apache.org

I have some doubts regarding message consumption and production, as well as 
transactional capabilities. I am using a Kafka template to produce a message 
within a transaction. After that, I execute another transaction that produces a 
message and intentionally throws a runtime exception to simulate a transaction 
rollback.

Next, I use the Kafka AdminClient to retrieve the latest offset for the topic 
partition and the consumer group's offsets for the same topic partition. 
However, when I compare the offset numbers, I notice a difference. In this 
example, the consumer has 4 offsets, while the topic has only 2.

I have come across references to this issue in a Spring-Kafka report, 
specifically in the Kafka-10683 report, where developers describe it as either 
Bogus or Pseudo Lag.

I am keen on resolving this problem, and I would greatly appreciate hearing 
about your experiences and knowledge regarding this matter.

Thank you very much
Henry

[jira] [Created] (KAFKA-15130) Delete remote segments when delete a topic

[Lan Ding (Jira)]

Lan Ding created KAFKA-15130:


 Summary: Delete remote segments when delete a topic
 Key: KAFKA-15130
 URL: https://issues.apache.org/jira/browse/KAFKA-15130
 Project: Kafka
  Issue Type: Improvement
  Components: core
Affects Versions: 3.5.0, 3.4.0
Reporter: Lan Ding
Assignee: Lan Ding


When tired storage is enabled and {{delete.topic.enable=true}} , deleting a 
topic should also delete the corresponding segments of that topic on the remote 
system, and cancel the RLMTask for that topic.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-944 Support async runtimes in consumer

[Erik van Oosten]

It seems KIP-941 was already taken. Updated to: KIP-944.


Op 28-06-2023 om 10:11 schreef Erik van Oosten:

Hello developers of the Java based consumer,

I submitted https://github.com/apache/kafka/pull/13914 to fix a long 
standing problem that the Kafka consumer on the JVM is not usable from 
asynchronous runtimes such as Kotlin co-routines and ZIO. However, 
since it extends the public API I was requested to create a KIP.


So here it is:
KIP-944 Support async runtimes in consumer 
https://cwiki.apache.org/confluence/x/chw0Dw


Any questions, comments, ideas and other additions are welcome!

The KIP should be complete except for the testing section. As far as I 
am aware there are no tests for the current behavior. Any help in this 
area would be appreciated.


Kind regards,
    Erik.


--
Erik van Oosten
e.vanoos...@grons.nl
https://day-to-day-stuff.blogspot.com

[DISCUSS] KIP-941 Support async runtimes in consumer

[Erik van Oosten]

Hello developers of the Java based consumer,

I submitted https://github.com/apache/kafka/pull/13914 to fix a long 
standing problem that the Kafka consumer on the JVM is not usable from 
asynchronous runtimes such as Kotlin co-routines and ZIO. However, since 
it extends the public API I was requested to create a KIP.


So here it is:
KIP-941 Support async runtimes in consumer 
https://cwiki.apache.org/confluence/x/chw0Dw


Any questions, comments, ideas and other additions are welcome!

The KIP should be complete except for the testing section. As far as I 
am aware there are no tests for the current behavior. Any help in this 
area would be appreciated.


Kind regards,
    Erik.


--
Erik van Oosten
e.vanoos...@grons.nl
https://day-to-day-stuff.blogspot.com

[jira] [Created] (KAFKA-15129) Clean up all metrics that were forgotten to be closed

[hudeqi (Jira)]

hudeqi created KAFKA-15129:
--

 Summary: Clean up all metrics that were forgotten to be closed
 Key: KAFKA-15129
 URL: https://issues.apache.org/jira/browse/KAFKA-15129
 Project: Kafka
  Issue Type: Improvement
  Components: controller, core, log
Affects Versions: 3.5.0
Reporter: hudeqi
Assignee: hudeqi






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Apply to contribute to the Kafka project

[Josep Prat]

Hi there!

Thanks for your interest in contributing to Apache Kafka, you should be all
set now. Let me know if there is any problem.

Best,

On Wed, Jun 28, 2023 at 9:47 AM isding_l  wrote:

>
> Hi，
>
>
> I'd like to request permissions to contribute to Kafka to propose a KIP.
>
>
> Wiki ID: isDing_L
> Kira ID: isDing_L
>
>
> Thank you
>
>
> --
> 发自我的网易邮箱手机智能版



-- 
[image: Aiven] 

*Josep Prat*
Open Source Engineering Director, *Aiven*
josep.p...@aiven.io   |   +491715557497
aiven.io    |   
     
*Aiven Deutschland GmbH*
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
Amtsgericht Charlottenburg, HRB 209739 B

Apply to contribute to the Kafka project

[isding_l]

Hi，


I'd like to request permissions to contribute to Kafka to propose a KIP.


Wiki ID: isDing_L
Kira ID: isDing_L


Thank you


--
发自我的网易邮箱手机智能版