Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1540

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 734433 lines...]
[2023-01-25T05:50:55.087Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 170 > MirrorMakerIntegrationTest > 
testCommitOffsetsThrowTimeoutException(String) > 
kafka.tools.MirrorMakerIntegrationTest.testCommitOffsetsThrowTimeoutException(String)[1]
 PASSED
[2023-01-25T05:50:55.087Z] 
[2023-01-25T05:50:55.087Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 170 > MirrorMakerIntegrationTest > 
testCommitOffsetsThrowTimeoutException(String) > 
kafka.tools.MirrorMakerIntegrationTest.testCommitOffsetsThrowTimeoutException(String)[2]
 STARTED
[2023-01-25T05:50:57.037Z] 
[2023-01-25T05:50:57.037Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 170 > MirrorMakerIntegrationTest > 
testCommitOffsetsThrowTimeoutException(String) > 
kafka.tools.MirrorMakerIntegrationTest.testCommitOffsetsThrowTimeoutException(String)[2]
 PASSED
[2023-01-25T05:50:57.037Z] 
[2023-01-25T05:50:57.037Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 170 > ReplicationUtilsTest > testUpdateLeaderAndIsr() STARTED
[2023-01-25T05:50:57.037Z] 
[2023-01-25T05:50:57.037Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 170 > ReplicationUtilsTest > testUpdateLeaderAndIsr() PASSED
[2023-01-25T05:50:57.037Z] 
[2023-01-25T05:50:57.037Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 170 > ZooKeeperClientTest > testZNodeChangeHandlerForDataChange() 
STARTED
[2023-01-25T05:50:57.037Z] 
[2023-01-25T05:50:57.037Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 170 > ZooKeeperClientTest > testZNodeChangeHandlerForDataChange() 
PASSED
[2023-01-25T05:50:57.037Z] 
[2023-01-25T05:50:57.037Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 170 > ZooKeeperClientTest > testZooKeeperSessionStateMetric() STARTED
[2023-01-25T05:50:57.037Z] 
[2023-01-25T05:50:57.037Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 170 > ZooKeeperClientTest > testZooKeeperSessionStateMetric() PASSED
[2023-01-25T05:50:57.037Z] 
[2023-01-25T05:50:57.037Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 170 > ZooKeeperClientTest > testExceptionInBeforeInitializingSession() 
STARTED
[2023-01-25T05:50:58.058Z] 
[2023-01-25T05:50:58.058Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 170 > ZooKeeperClientTest > testExceptionInBeforeInitializingSession() 
PASSED
[2023-01-25T05:50:58.058Z] 
[2023-01-25T05:50:58.058Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 170 > ZooKeeperClientTest > testGetChildrenExistingZNode() STARTED
[2023-01-25T05:50:58.058Z] 
[2023-01-25T05:50:58.058Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 170 > ZooKeeperClientTest > testGetChildrenExistingZNode() PASSED
[2023-01-25T05:50:58.058Z] 
[2023-01-25T05:50:58.058Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 170 > ZooKeeperClientTest > testConnection() STARTED
[2023-01-25T05:50:58.058Z] 
[2023-01-25T05:50:58.058Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 170 > ZooKeeperClientTest > testConnection() PASSED
[2023-01-25T05:50:58.058Z] 
[2023-01-25T05:50:58.058Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 170 > ZooKeeperClientTest > testZNodeChangeHandlerForCreation() STARTED
[2023-01-25T05:50:58.739Z] 
[2023-01-25T05:50:58.739Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 172 > BrokerRegistrationRequestTest > 
testRegisterZkWithKRaftMigrationEnabled(ClusterInstance) > 
unit.kafka.server.BrokerRegistrationRequestTest.testRegisterZkWithKRaftMigrationEnabled(ClusterInstance)[1]
 PASSED
[2023-01-25T05:50:58.739Z] 
[2023-01-25T05:50:58.739Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 172 > BrokerRegistrationRequestTest > 
testRegisterZkWithKRaftOldMetadataVersion(ClusterInstance) > 
unit.kafka.server.BrokerRegistrationRequestTest.testRegisterZkWithKRaftOldMetadataVersion(ClusterInstance)[1]
 STARTED
[2023-01-25T05:50:59.249Z] 
[2023-01-25T05:50:59.249Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 170 > ZooKeeperClientTest > testZNodeChangeHandlerForCreation() PASSED
[2023-01-25T05:50:59.249Z] 
[2023-01-25T05:50:59.249Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 170 > ZooKeeperClientTest > testGetAclExistingZNode() STARTED
[2023-01-25T05:50:59.249Z] 
[2023-01-25T05:50:59.249Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 170 > ZooKeeperClientTest > testGetAclExistingZNode() PASSED
[2023-01-25T05:50:59.249Z] 
[2023-01-25T05:50:59.249Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 170 > ZooKeeperClientTest > testSessionExpiryDuringClose() STARTED
[2023-01-25T05:50:59.249Z] 
[2023-01-25T05:50:59.249Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 170 > ZooKeeperClientTest > 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.4 #65

[Apache Jenkins Server]

See 

[DISCUSS] KIP-901: Add connectorDeleted flag when stopping tasks

[Hector Geraldino (BLOOMBERG/ 919 3RD A)]

Hi everyone,

I've submitted KIP-901, which adds an overloaded Task#stop(boolean 
connectorDeleted) method to the public Kafka Connect APIs:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-901%3A+Add+connectorDeleted+flag+when+stopping+tasks

This KIP can be seen as a companion (or counterpart) of KIP-883, which aims to 
provide the same feature but at the Connector level (there's a separate 
discussion thread on this list). The KIP also supersedes "KIP-419: Safely 
notify Kafka Connect SourceTask is stopped". 

The main goal is to let the task being stopped that it is due to the connector 
being deleted, so the task can perform any cleanup actions as part of the 
connector's deletion process.

I look forward for your feedback and comments.

Thanks!
Hector

[jira] [Created] (KAFKA-14651) Add connectorDeleted flag when stopping tasks

[Hector Geraldino (Jira)]

Hector Geraldino created KAFKA-14651:


 Summary: Add connectorDeleted flag when stopping tasks
 Key: KAFKA-14651
 URL: https://issues.apache.org/jira/browse/KAFKA-14651
 Project: Kafka
  Issue Type: Improvement
  Components: KafkaConnect
Reporter: Hector Geraldino
Assignee: Hector Geraldino


It would be useful for Connectors to know when its instance is being deleted. 
This will give a chance to connectors to perform any cleanup tasks (e.g. 
deleting external resources, or deleting offsets) before the connector is 
completely removed from the cluster.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1539

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 526870 lines...]
[2023-01-25T02:38:33.506Z] 
[2023-01-25T02:38:33.506Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 174 > KafkaZkClientTest > testIsrChangeNotificationGetters() PASSED
[2023-01-25T02:38:33.506Z] 
[2023-01-25T02:38:33.506Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 174 > KafkaZkClientTest > testLogDirEventNotificationsDeletion() 
STARTED
[2023-01-25T02:38:33.506Z] 
[2023-01-25T02:38:33.506Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 174 > KafkaZkClientTest > testLogDirEventNotificationsDeletion() PASSED
[2023-01-25T02:38:33.506Z] 
[2023-01-25T02:38:33.506Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 174 > KafkaZkClientTest > testGetLogConfigs() STARTED
[2023-01-25T02:38:34.615Z] 
[2023-01-25T02:38:34.615Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 174 > KafkaZkClientTest > testGetLogConfigs() PASSED
[2023-01-25T02:38:34.615Z] 
[2023-01-25T02:38:34.615Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 174 > KafkaZkClientTest > testBrokerSequenceIdMethods() STARTED
[2023-01-25T02:38:34.615Z] 
[2023-01-25T02:38:34.615Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 174 > KafkaZkClientTest > testBrokerSequenceIdMethods() PASSED
[2023-01-25T02:38:34.615Z] 
[2023-01-25T02:38:34.615Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 174 > KafkaZkClientTest > testAclMethods() STARTED
[2023-01-25T02:38:34.615Z] 
[2023-01-25T02:38:34.615Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 174 > KafkaZkClientTest > testAclMethods() PASSED
[2023-01-25T02:38:34.615Z] 
[2023-01-25T02:38:34.615Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 174 > KafkaZkClientTest > testCreateSequentialPersistentPath() STARTED
[2023-01-25T02:38:34.615Z] 
[2023-01-25T02:38:34.615Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 174 > KafkaZkClientTest > testCreateSequentialPersistentPath() PASSED
[2023-01-25T02:38:34.615Z] 
[2023-01-25T02:38:34.615Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 174 > KafkaZkClientTest > testConditionalUpdatePath() STARTED
[2023-01-25T02:38:35.724Z] 
[2023-01-25T02:38:35.724Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 174 > KafkaZkClientTest > testConditionalUpdatePath() PASSED
[2023-01-25T02:38:35.724Z] 
[2023-01-25T02:38:35.724Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 174 > KafkaZkClientTest > testGetAllTopicsInClusterTriggersWatch() 
STARTED
[2023-01-25T02:38:35.724Z] 
[2023-01-25T02:38:35.724Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 174 > KafkaZkClientTest > testGetAllTopicsInClusterTriggersWatch() 
PASSED
[2023-01-25T02:38:35.724Z] 
[2023-01-25T02:38:35.724Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 174 > KafkaZkClientTest > testDeleteTopicZNode() STARTED
[2023-01-25T02:38:35.724Z] 
[2023-01-25T02:38:35.724Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 174 > KafkaZkClientTest > testDeleteTopicZNode() PASSED
[2023-01-25T02:38:35.724Z] 
[2023-01-25T02:38:35.724Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 174 > KafkaZkClientTest > testDeletePath() STARTED
[2023-01-25T02:38:36.833Z] 
[2023-01-25T02:38:36.833Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 174 > KafkaZkClientTest > testDeletePath() PASSED
[2023-01-25T02:38:36.833Z] 
[2023-01-25T02:38:36.833Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 174 > KafkaZkClientTest > testGetBrokerMethods() STARTED
[2023-01-25T02:38:36.833Z] 
[2023-01-25T02:38:36.833Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 174 > KafkaZkClientTest > testGetBrokerMethods() PASSED
[2023-01-25T02:38:36.833Z] 
[2023-01-25T02:38:36.833Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 174 > KafkaZkClientTest > testJuteMaxBufffer() STARTED
[2023-01-25T02:38:36.833Z] 
[2023-01-25T02:38:36.833Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 174 > KafkaZkClientTest > testJuteMaxBufffer() PASSED
[2023-01-25T02:38:36.833Z] 
[2023-01-25T02:38:36.833Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 174 > KafkaZkClientTest > testCreateTokenChangeNotification() STARTED
[2023-01-25T02:38:37.772Z] 
[2023-01-25T02:38:37.772Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 174 > KafkaZkClientTest > testCreateTokenChangeNotification() PASSED
[2023-01-25T02:38:37.772Z] 
[2023-01-25T02:38:37.772Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 174 > KafkaZkClientTest > testGetTopicsAndPartitions() STARTED
[2023-01-25T02:38:37.772Z] 
[2023-01-25T02:38:37.772Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 174 > KafkaZkClientTest > testGetTopicsAndPartitions() PASSED

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.4 #64

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-14650) IQv2 can throw ConcurrentModificationException when accessing Tasks

[A. Sophie Blee-Goldman (Jira)]

A. Sophie Blee-Goldman created KAFKA-14650:
--

 Summary: IQv2 can throw ConcurrentModificationException when 
accessing Tasks 
 Key: KAFKA-14650
 URL: https://issues.apache.org/jira/browse/KAFKA-14650
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 3.4.0
Reporter: A. Sophie Blee-Goldman


>From failure in *[PositionRestartIntegrationTest.verifyStore[cache=false, 
>log=true, supplier=IN_MEMORY_WINDOW, 
>kind=PAPI]|https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.4/63/testReport/junit/org.apache.kafka.streams.integration/PositionRestartIntegrationTest/Build___JDK_11_and_Scala_2_13___verifyStore_cache_false__log_true__supplier_IN_MEMORY_WINDOW__kind_PAPI_/]*
java.util.ConcurrentModificationException
at 
java.base/java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1208)
at java.base/java.util.TreeMap$EntryIterator.next(TreeMap.java:1244)
at java.base/java.util.TreeMap$EntryIterator.next(TreeMap.java:1239)
at java.base/java.util.HashMap.putMapEntries(HashMap.java:508)
at java.base/java.util.HashMap.putAll(HashMap.java:781)
at 
org.apache.kafka.streams.processor.internals.Tasks.allTasksPerId(Tasks.java:361)
at 
org.apache.kafka.streams.processor.internals.TaskManager.allTasks(TaskManager.java:1537)
at 
org.apache.kafka.streams.processor.internals.StreamThread.allTasks(StreamThread.java:1278)
at org.apache.kafka.streams.KafkaStreams.query(KafkaStreams.java:1921)
at 
org.apache.kafka.streams.integration.utils.IntegrationTestUtils.iqv2WaitForResult(IntegrationTestUtils.java:168)
at 
org.apache.kafka.streams.integration.PositionRestartIntegrationTest.shouldReachExpectedPosition(PositionRestartIntegrationTest.java:438)
at 
org.apache.kafka.streams.integration.PositionRestartIntegrationTest.verifyStore(PositionRestartIntegrationTest.java:423)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[ANNOUNCE] Share your Streaming Stories with us at Current 2023

[Israel Ekpo]

Do you have a great data streaming story to share?

We want to hear from you!

Speaking at Current 2023 is a great way to connect with hundreds of your
peers, become more involved in the data streaming community, and have a
public platform for you to share your story of the future of streaming and
real-time data.

Check out the details at the link below and share your CFP with us

https://sessionize.com/current-2023/

Important Dates:

Call for Papers opens: January 12, 2023

Speaker Office Hours:
Wednesday, February 1: 10:00 - 11:00 GMT and  11:00 - 12:00 PST
Wednesday, March 8: 10:00 - 11:00 GMT and 11:00 - 12:00 PST
Wednesday, March 15: 10:00 - 11:00 GMT and 11:00 - 12:00 PST

Call for Papers closes: March 20, 2023

Speaker Notifications sent: April 18, 2023

Questions? Please email speak...@currentevent.io

We are waiting to hear from you.

Jenkins build is unstable: Kafka » Kafka Branch Builder » 3.4 #63

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-14649) Failures instantiating Connect plugins crash worker on startup

[Greg Harris (Jira)]

Greg Harris created KAFKA-14649:
---

 Summary: Failures instantiating Connect plugins crash worker on 
startup
 Key: KAFKA-14649
 URL: https://issues.apache.org/jira/browse/KAFKA-14649
 Project: Kafka
  Issue Type: Bug
  Components: KafkaConnect
Affects Versions: 3.3.0, 3.2.0, 3.0.0, 3.1.0, 3.4.0
Reporter: Greg Harris
Assignee: Greg Harris


Connect plugin path scanning evaluates the version() method of plugins to 
determine which version of a plugin to load, and what version to advertise as 
part of the REST API. This process involves reflectively constructing an 
instance of the class and calling the version method, which can fail in the 
following scenarios:

1. If a plugin throws an exception from a static initialization block
2. If a plugin does not have a default constructor (such as a non-static inner 
class)
3. If a plugin has a default constructor is not public
4. If a plugin throws an exception from the default constructor
5. If a plugin's version method throws an exception

If any of the above is true for any single connector or rest extension on the 
classpath or plugin.path, the worker will fail to start up entirely. This is 
primarily an issue in development and test environments, because they are 
easy-to-make code mistakes that would generally not make it to a release.

It is desirable for the worker to instead log these exceptions and continue. 
This will prevent one mis-implemented plugin from affecting other plugins, 
while still causing integration tests to fail against the plugin itself. We can 
augment logging to make it clear how to correct these failures, where before it 
was rather opaque and difficult to debug.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1538

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 354183 lines...]
[2023-01-24T22:00:31.733Z] > Task :connect:json:compileJava UP-TO-DATE
[2023-01-24T22:00:31.733Z] > Task :connect:json:classes UP-TO-DATE
[2023-01-24T22:00:31.733Z] > Task :connect:json:javadoc SKIPPED
[2023-01-24T22:00:31.733Z] > Task :connect:json:javadocJar
[2023-01-24T22:00:31.733Z] > Task :connect:api:copyDependantLibs UP-TO-DATE
[2023-01-24T22:00:31.733Z] > Task :connect:api:jar UP-TO-DATE
[2023-01-24T22:00:31.733Z] > Task 
:connect:api:generateMetadataFileForMavenJavaPublication
[2023-01-24T22:00:31.733Z] > Task :connect:json:compileTestJava UP-TO-DATE
[2023-01-24T22:00:31.733Z] > Task :connect:json:copyDependantLibs UP-TO-DATE
[2023-01-24T22:00:31.733Z] > Task :connect:json:jar UP-TO-DATE
[2023-01-24T22:00:31.733Z] > Task 
:connect:json:generateMetadataFileForMavenJavaPublication
[2023-01-24T22:00:31.733Z] > Task :connect:json:testClasses UP-TO-DATE
[2023-01-24T22:00:31.733Z] > Task :connect:json:testJar
[2023-01-24T22:00:31.733Z] > Task :connect:json:testSrcJar
[2023-01-24T22:00:31.733Z] > Task 
:connect:json:publishMavenJavaPublicationToMavenLocal
[2023-01-24T22:00:31.733Z] > Task :connect:json:publishToMavenLocal
[2023-01-24T22:00:37.342Z] > Task :connect:api:javadoc
[2023-01-24T22:00:37.342Z] > Task :connect:api:javadocJar
[2023-01-24T22:00:37.342Z] > Task :connect:api:compileTestJava UP-TO-DATE
[2023-01-24T22:00:37.342Z] > Task :connect:api:processTestResources NO-SOURCE
[2023-01-24T22:00:37.342Z] > Task :connect:api:testClasses UP-TO-DATE
[2023-01-24T22:00:37.342Z] > Task :connect:api:testJar
[2023-01-24T22:00:38.269Z] > Task :connect:api:testSrcJar
[2023-01-24T22:00:38.269Z] > Task 
:connect:api:publishMavenJavaPublicationToMavenLocal
[2023-01-24T22:00:38.269Z] > Task :connect:api:publishToMavenLocal
[2023-01-24T22:00:38.269Z] 
[2023-01-24T22:00:38.269Z] > Task :clients:javadoc
[2023-01-24T22:00:38.269Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/OAuthBearerLoginCallbackHandler.java:151:
 warning - Tag @link: reference not found: 
[2023-01-24T22:00:39.282Z] 
[2023-01-24T22:00:39.282Z] > Task :streams:javadoc
[2023-01-24T22:00:39.282Z] > Task :streams:javadocJar
[2023-01-24T22:00:39.282Z] > Task :streams:processTestResources UP-TO-DATE
[2023-01-24T22:00:41.308Z] 
[2023-01-24T22:00:41.308Z] > Task :clients:javadoc
[2023-01-24T22:00:41.308Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/secured/package-info.java:21:
 warning - Tag @link: reference not found: 
org.apache.kafka.common.security.oauthbearer
[2023-01-24T22:00:42.319Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/secured/package-info.java:21:
 warning - Tag @link: reference not found: 
org.apache.kafka.common.security.oauthbearer
[2023-01-24T22:00:42.319Z] 3 warnings
[2023-01-24T22:00:42.319Z] 
[2023-01-24T22:00:42.319Z] > Task :clients:javadocJar
[2023-01-24T22:00:44.513Z] 
[2023-01-24T22:00:44.513Z] > Task :clients:srcJar
[2023-01-24T22:00:44.513Z] Execution optimizations have been disabled for task 
':clients:srcJar' to ensure correctness due to the following reasons:
[2023-01-24T22:00:44.513Z]   - Gradle detected a problem with the following 
location: 
'/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/clients/src/generated/java'.
 Reason: Task ':clients:srcJar' uses this output of task 
':clients:processMessages' without declaring an explicit or implicit 
dependency. This can lead to incorrect results being produced, depending on 
what order the tasks are executed. Please refer to 
https://docs.gradle.org/7.6/userguide/validation_problems.html#implicit_dependency
 for more details about this problem.
[2023-01-24T22:00:45.525Z] 
[2023-01-24T22:00:45.525Z] > Task :clients:testJar
[2023-01-24T22:00:45.525Z] > Task :clients:testSrcJar
[2023-01-24T22:00:45.525Z] > Task 
:clients:publishMavenJavaPublicationToMavenLocal
[2023-01-24T22:00:45.525Z] > Task :clients:publishToMavenLocal
[2023-01-24T22:01:02.158Z] > Task :core:compileScala
[2023-01-24T22:02:04.109Z] > Task :core:classes
[2023-01-24T22:02:04.109Z] > Task :core:compileTestJava NO-SOURCE
[2023-01-24T22:02:33.334Z] > Task :core:compileTestScala
[2023-01-24T22:03:16.569Z] > Task :core:testClasses
[2023-01-24T22:03:16.569Z] > Task :streams:compileTestJava UP-TO-DATE
[2023-01-24T22:03:16.569Z] > Task :streams:testClasses UP-TO-DATE
[2023-01-24T22:03:17.477Z] > Task :streams:testJar
[2023-01-24T22:03:17.477Z] > Task :streams:testSrcJar
[2023-01-24T22:03:17.477Z] > Task 
:streams:publishMavenJavaPublicationToMavenLocal
[2023-01-24T22:03:17.477Z] > Task :streams:publishToMavenLocal
[2023-01-24T22:03:17.477Z] 
[2023-01-24T22:03:17.477Z] Deprecated Gradle features were used 

[jira] [Created] (KAFKA-14648) Do not fail clients if bootstrap servers is not immediately resolvable

[Jason Gustafson (Jira)]

Jason Gustafson created KAFKA-14648:
---

 Summary: Do not fail clients if bootstrap servers is not 
immediately resolvable
 Key: KAFKA-14648
 URL: https://issues.apache.org/jira/browse/KAFKA-14648
 Project: Kafka
  Issue Type: Bug
Reporter: Jason Gustafson


In dynamic environments, such as system tests, there is sometimes a delay 
between when a client is initialized and when the configured bootstrap servers 
become available in DNS. Currently clients will fail immediately if none of the 
bootstrap servers can resolve. It would be more convenient for these 
environments to provide a grace period to give more time for initialization. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-869: Improve Streams State Restoration Visibility

[Guozhang Wang]

Hi Matthias:

re "paused" -> "suspended": I got your point now, thanks. Just to
clarify the two functions are a bit different: "paused" tasks are
because of the topology being paused, i.e. from KIP-834; whereas
"suspended" tasks are when a restoring tasks are being removed before
it completes due to a follow-up rebalance, and this is to distinguish
with "onRestoreEnd", as described in KAFKA-10575. A suspended task is
no longer owned by the thread and hence there's no need to measure the
number of such tasks.

re: "restore-ratio": that's a good point. I like it to function in the
same way as the "records-rate" metrics. Will update the wiki.

re: making "restore-remaining-records-total" at INFO level: sounds
good to me too. I will also update the metric name a bit to be more
specific.



On Thu, Jan 19, 2023 at 2:35 PM Guozhang Wang
 wrote:
>
> Hello Matthias,
>
> Thanks for the feedback. I was on vacation for a while. Pardon for the
> late replies. Please see them inline below
>
> On Thu, Dec 1, 2022 at 11:23 PM Matthias J. Sax  wrote:
> >
> > Seems I am late to the party... Great KIP. Couple of questions from my side:
> >
> > (1) What is the purpose of `standby-updating-tasks`? It seems to be the
> > same as the number of assigned standby task? Not sure how useful it
> > would be?
> >
> In general, yes, it is the number of assigned standby tasks --- there
> will be transit times when the assigned standby tasks are not yet
> being updated but it would not last long --- but we do not yet have a
> direct gauge to expose this before, and users have to infer this from
> other indirect metrics.
>
> >
> >
> > (2) `active-paused-tasks` / `standby-paused-tasks` -- what does "paused"
> > exactly mean? There was a discussion about renaming the callback method
> > from pause to suspended. So should this be called `suspended`, too? And
> > if yes, how is it useful for users?
> >
> Pausing here refers to "KIP-834: Pause / Resume KafkaStreams
> Topologies" 
> (https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211882832).
> When a topology is paused, all its tasks including standbys will be
> paused too.
>
> I'm not aware of a discussion to rename the call name to "suspend" for
> KIP-834. Could you point me to the reference?
>
> >
> >
> > (3) `restore-ratio`: the description says
> >
> > > The fraction of time the thread spent on restoring active or standby tasks
> >
> > I find the term "restoring" does only apply to active tasks, but not to
> > standbys. Can we reword this?
> >
> Yeah I have been discussing this with others in the community a bit as
> well, but so far I have not been convinced of a better name than it.
> Some other alternatives being discussed but not win everyone's love is
> "restore-or-update-ratio", "process-ratio" (for the restore thread
> that means restoring or updating), and "io-ratio".
>
> The only one so far that I feel is probably better, is
> "state-update-ratio". If folks feel this one is better than
> "restore-ratio" I'm happy to update.
>
> >
> > (4) `restore-call-rate`: not sure what you exactly mean by "restore calls"?
> >
> This is similar to the "io-calls-rate" in the selector classes, i.e.
> the number of "restore" function calls made. It's argurably a very
> low-level metrics but I included it since it could be useful in some
> debugging scenarios.
>
> >
> > (5) `restore-remaining-records-total` -- why is this a task metric?
> > Seems we could roll it up into a thread metric that we report at INFO
> > level (we could still have per-task DEBUG level metric for it in addition).
> >
> The rationale behind it is the general principle in metrics design
> that "Kafka would provide the lowest necessary metrics levels, and
> users can do the roll-ups however they want".
>
> >
> > (6) What about "warmup tasks"? Internally, we treat them as standbys,
> > but it seems it's hard for users to reason about it in the scale-out
> > warm-up case. Would it be helpful (and possible) to report "warmup
> > progress" explicitly?
> >
> At the restore thread level, we cannot differentiate standby tasks
> from warmup tasks since the latter is created exactly just like the
> former. But I do agree this is an issue for visibility that worth
> addressing, I think another KIP would be needed to first consider
> distinguishing these two at the class level.
>
> >
> > -Matthias
> >
> >
> > On 11/1/22 2:44 AM, Lucas Brutschy wrote:
> > > We need this!
> > >
> > > + 1 non binding
> > >
> > > Cheers,
> > > Lucas
> > >
> > > On Tue, Nov 1, 2022 at 10:01 AM Bruno Cadonna  wrote:
> > >>
> > >> Guozhang,
> > >>
> > >> Thanks for the KIP!
> > >>
> > >> +1 (binding)
> > >>
> > >> Best,
> > >> Bruno
> > >>
> > >> On 25.10.22 22:07, Walker Carlson wrote:
> > >>> +1 non binding
> > >>>
> > >>> Thanks for the kip!
> > >>>
> > >>> On Thu, Oct 20, 2022 at 10:25 PM John Roesler  
> > >>> wrote:
> > >>>
> >  Thanks for the KIP, Guozhang!
> > 
> >  I'm +1 (binding)
> > 
> >  -John
> > 
> >  

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.4 #62

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 522882 lines...]
[2023-01-24T14:49:50.304Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 173 > KafkaZkClientTest > testGetAllTopicsInClusterTriggersWatch() 
STARTED
[2023-01-24T14:49:50.304Z] 
[2023-01-24T14:49:50.304Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 173 > KafkaZkClientTest > testGetAllTopicsInClusterTriggersWatch() 
PASSED
[2023-01-24T14:49:50.304Z] 
[2023-01-24T14:49:50.304Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 173 > KafkaZkClientTest > testDeleteTopicZNode() STARTED
[2023-01-24T14:49:51.418Z] 
[2023-01-24T14:49:51.418Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 173 > KafkaZkClientTest > testDeleteTopicZNode() PASSED
[2023-01-24T14:49:51.418Z] 
[2023-01-24T14:49:51.418Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 173 > KafkaZkClientTest > testDeletePath() STARTED
[2023-01-24T14:49:51.418Z] 
[2023-01-24T14:49:51.418Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 173 > KafkaZkClientTest > testDeletePath() PASSED
[2023-01-24T14:49:51.418Z] 
[2023-01-24T14:49:51.418Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 173 > KafkaZkClientTest > testGetBrokerMethods() STARTED
[2023-01-24T14:49:51.418Z] 
[2023-01-24T14:49:51.418Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 173 > KafkaZkClientTest > testGetBrokerMethods() PASSED
[2023-01-24T14:49:51.418Z] 
[2023-01-24T14:49:51.418Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 173 > KafkaZkClientTest > testJuteMaxBufffer() STARTED
[2023-01-24T14:49:52.361Z] 
[2023-01-24T14:49:52.361Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 173 > KafkaZkClientTest > testJuteMaxBufffer() PASSED
[2023-01-24T14:49:52.361Z] 
[2023-01-24T14:49:52.361Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 173 > KafkaZkClientTest > testCreateTokenChangeNotification() STARTED
[2023-01-24T14:49:52.361Z] 
[2023-01-24T14:49:52.361Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 173 > KafkaZkClientTest > testCreateTokenChangeNotification() PASSED
[2023-01-24T14:49:52.361Z] 
[2023-01-24T14:49:52.361Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 173 > KafkaZkClientTest > testGetTopicsAndPartitions() STARTED
[2023-01-24T14:49:52.361Z] 
[2023-01-24T14:49:52.361Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 173 > KafkaZkClientTest > testGetTopicsAndPartitions() PASSED
[2023-01-24T14:49:52.361Z] 
[2023-01-24T14:49:52.361Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 173 > KafkaZkClientTest > testChroot(boolean) > 
kafka.zk.KafkaZkClientTest.testChroot(boolean)[1] STARTED
[2023-01-24T14:49:53.398Z] 
[2023-01-24T14:49:53.398Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 173 > KafkaZkClientTest > testChroot(boolean) > 
kafka.zk.KafkaZkClientTest.testChroot(boolean)[1] PASSED
[2023-01-24T14:49:53.398Z] 
[2023-01-24T14:49:53.398Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 173 > KafkaZkClientTest > testChroot(boolean) > 
kafka.zk.KafkaZkClientTest.testChroot(boolean)[2] STARTED
[2023-01-24T14:49:53.398Z] 
[2023-01-24T14:49:53.398Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 173 > KafkaZkClientTest > testChroot(boolean) > 
kafka.zk.KafkaZkClientTest.testChroot(boolean)[2] PASSED
[2023-01-24T14:49:53.398Z] 
[2023-01-24T14:49:53.398Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 173 > KafkaZkClientTest > testRegisterBrokerInfo() STARTED
[2023-01-24T14:49:54.365Z] 
[2023-01-24T14:49:54.366Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 174 > AllocateProducerIdsRequestTest > 
testAllocateProducersIdSentToNonController() > 
unit.kafka.server.AllocateProducerIdsRequestTest.testAllocateProducersIdSentToNonController()[1]
 PASSED
[2023-01-24T14:49:54.538Z] 
[2023-01-24T14:49:54.538Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 173 > KafkaZkClientTest > testRegisterBrokerInfo() PASSED
[2023-01-24T14:49:54.538Z] 
[2023-01-24T14:49:54.538Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 173 > KafkaZkClientTest > testRetryRegisterBrokerInfo() STARTED
[2023-01-24T14:49:54.538Z] 
[2023-01-24T14:49:54.538Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 173 > KafkaZkClientTest > testRetryRegisterBrokerInfo() PASSED
[2023-01-24T14:49:54.538Z] 
[2023-01-24T14:49:54.538Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 173 > KafkaZkClientTest > testConsumerOffsetPath() STARTED
[2023-01-24T14:49:54.538Z] 
[2023-01-24T14:49:54.538Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 173 > KafkaZkClientTest > testConsumerOffsetPath() PASSED
[2023-01-24T14:49:54.538Z] 
[2023-01-24T14:49:54.538Z] Gradle Test Run :core:integrationTest > 

Re: [DISCUSS] KIP-890 Server Side Defense

[Justine Olshan]

Hey Matthias,

I have actually never heard of KIP-280 so thanks for bringing it up. That
seems interesting. I wonder how it would work though -- would it build an
offset map with just the latest timestamp for a key? I wonder if ordering
assumptions are baked in there, why not use offset-based compaction.

I was also not aware of this "guarantee" with regards to broker side time.
I think that we can do in order handling for a given producer, but not
across all producers. However, we can't guarantee that anyway.

Let me know if you have any concerns here.

Thanks,
Justine

On Mon, Jan 23, 2023 at 6:33 PM Matthias J. Sax  wrote:

> Just a side note about Guozhang comments about timestamps.
>
> If the producer sets the timestamp, putting the record into purgatory
> seems not to be an issue (as already said: for this case we don't
> guarantee timestamp order between writes of different producers anyway).
> However, if the broker sets the timestamp, the expectation is that there
> is no out-of-order data in the partition ever; if we would introduce
> out-of-order data for this case (for interleaved writes of different
> producers), it seems we would violate the current contract? (To be fair:
> I don't know if that's an official contract, but I assume people rely on
> this behavior -- and it "advertised" in many public talks...)
>
> About compaction: there is actually KIP-280 that adds timestamp based
> compaction what is a very useful feature for Kafka Streams with regard
> to out-of-order data handling. So the impact if we introduce
> out-of-order data could be larger scoped.
>
>
> -Matthias
>
>
> On 1/20/23 4:48 PM, Justine Olshan wrote:
> > Hey Artem,
> >
> > I see there is a check for transactional producers. I'm wondering if we
> > don't handle the epoch overflow case. I'm also not sure it will be a huge
> > issue to extend to transactional producers, but maybe I'm missing
> something.
> >
> > As for the recovery path -- I think Guozhang's point was if we have a bad
> > client that repeatedly tries to produce without adding to the transaction
> > we would do the following:
> > a) if not fatal, we just fail the produce request over and over
> > b) if fatal, we fence the producer
> >
> > Here with B, the issue with the client would be made clear more quickly.
> I
> > suppose there are some intermediate cases where the issue only occurs
> > sometimes, but I wonder if we should consider how to recover with clients
> > who don't behave as expected anyway.
> >
> > I think there is a place for the abortable error that we are adding --
> just
> > abort and try again. But I think there are also some cases where trying
> to
> > recover overcomplicates some logic. Especially if we are considering
> older
> > clients -- there I'm not sure if there's a ton we can do besides fail the
> > batch or fence the producer. With newer clients, we can consider more
> > options for what can just be recovered after aborting. But epochs might
> be
> > a hard one unless we also want to reset producer ID.
> >
> > Thanks,
> > Justine
> >
> >
> >
> > On Fri, Jan 20, 2023 at 3:59 PM Artem Livshits
> >  wrote:
> >
> >>>   besides the poorly written client case
> >>
> >> A poorly written client could create a lot of grief to people who run
> Kafka
> >> brokers :-), so when deciding to make an error fatal I would see if
> there
> >> is a reasonable recovery path rather than how often it could happen.
> If we
> >> have solid implementation of transactions (which I hope we'll do as a
> >> result of this KIP), it would help to recover from a large class of
> errors
> >> by just aborting a transaction, even if the cause of error is a race
> >> condition or etc.
> >>
> >> -Artem
> >>
> >> On Fri, Jan 20, 2023 at 3:26 PM Justine Olshan
> >> 
> >> wrote:
> >>
> >>> Artem --
> >>> I guess the discussion path we were going down is when we expect to see
> >>> this error. I mentioned that it was hard to come up with cases for when
> >> the
> >>> producer would still be around to receive the error besides the poorly
> >>> written client case.
> >>> If we don't expect to have a producer to receive the response, it sort
> of
> >>> makes sense for it to be fatal.
> >>>
> >>> I had some discussion with Jason offline about the epoch being off
> cases
> >>> and I'm not sure we could find a ton (outside of produce requests)
> where
> >> we
> >>> could/should recover. I'd be happy to hear some examples though, maybe
> >> I'm
> >>> missing something.
> >>>
> >>> Thanks,
> >>> Justine
> >>>
> >>> On Fri, Jan 20, 2023 at 3:19 PM Artem Livshits
> >>>  wrote:
> >>>
>  In general, I'd like to avoid fatal errors as much as possible, in
> some
>  sense fatal errors just push out recovery logic to the application
> >> which
>  either complicates the application or leads to disruption (we've seen
> >>> cases
>  when a transient broker error could lead to work stoppage when
> >>> applications
>  need to be manually restarted).  I think we should 

Kafka running on Windows

[Shafiq Jetha]

Hi all,

There is an issue trying to get Kafka working correctly on Windows. Here is a 
link to the GitHub issue: https://github.com/apache/kafka/pull/12331

We will be ramping up our Kafka dev with a team split between Windows and Macs, 
so it would be good if we could get some movement on this or at least some 
ideas around how we can help get this working.

Thanks all,
 Shafiq.

Sent from Outlook for iOS

Re: [VOTE] KIP-869: Improve Streams State Restoration Visibility

[Walker Carlson]

Hey, I'm changing my vote to binding now :)

On Mon, Jan 23, 2023 at 9:38 PM Matthias J. Sax  wrote:

> Thanks Guozhang. Couple of clarifications and follow up questions.
>
>
> >> I'm not aware of a discussion to rename the call name to "suspend" for
> >> KIP-834. Could you point me to the reference?
>
> My commend was not about KIP-834, but about this KIP. You originally
> proposed to call the new call-back `onRestorePause` but to avoid
> confusion it was improved and renamed to `onRestoreSuspended`.
>
>
> > The only one so far that I feel is probably better, is
> > "state-update-ratio". If folks feel this one is better than
> > "restore-ratio" I'm happy to update.
>
> Could we actually report two metric, one for the restore phase
> (restore-ration), and one for steady state ([standby-]update-ratio)?
>
> I could like with `state-update-ratio` if we want to have a single
> metric for both, but splitting them sound useful to me.
>
>
> > (4) `restore-call-rate`
>
> Maybe we can clarify in the description a little bit. I agree it's very
> low level but if you think it could be useful to debugging, I have no
> objection.
>
>
> > The rationale behind it is the general principle in metrics design
> > that "Kafka would provide the lowest necessary metrics levels, and
> > users can do the roll-ups however they want".
>
> That's fair, but it seems to be a rather important metric, and having it
> only at DEBUG level seems not ideal? Could we make it INFO level, even
> if it's a task level (ie, apply an exception to the rule).
>
>
>
> -Matthias
>
>
>
> On 1/19/23 2:35 PM, Guozhang Wang wrote:
> > Hello Matthias,
> >
> > Thanks for the feedback. I was on vacation for a while. Pardon for the
> > late replies. Please see them inline below
> >
> > On Thu, Dec 1, 2022 at 11:23 PM Matthias J. Sax 
> wrote:
> >>
> >> Seems I am late to the party... Great KIP. Couple of questions from my
> side:
> >>
> >> (1) What is the purpose of `standby-updating-tasks`? It seems to be the
> >> same as the number of assigned standby task? Not sure how useful it
> >> would be?
> >>
> > In general, yes, it is the number of assigned standby tasks --- there
> > will be transit times when the assigned standby tasks are not yet
> > being updated but it would not last long --- but we do not yet have a
> > direct gauge to expose this before, and users have to infer this from
> > other indirect metrics.
> >
> >>
> >>
> >> (2) `active-paused-tasks` / `standby-paused-tasks` -- what does "paused"
> >> exactly mean? There was a discussion about renaming the callback method
> >> from pause to suspended. So should this be called `suspended`, too? And
> >> if yes, how is it useful for users?
> >>
> > Pausing here refers to "KIP-834: Pause / Resume KafkaStreams
> > Topologies" (
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211882832
> ).
> > When a topology is paused, all its tasks including standbys will be
> > paused too.
> >
> > I'm not aware of a discussion to rename the call name to "suspend" for
> > KIP-834. Could you point me to the reference?
> >
> >>
> >>
> >> (3) `restore-ratio`: the description says
> >>
> >>> The fraction of time the thread spent on restoring active or standby
> tasks
> >>
> >> I find the term "restoring" does only apply to active tasks, but not to
> >> standbys. Can we reword this?
> >>
> > Yeah I have been discussing this with others in the community a bit as
> > well, but so far I have not been convinced of a better name than it.
> > Some other alternatives being discussed but not win everyone's love is
> > "restore-or-update-ratio", "process-ratio" (for the restore thread
> > that means restoring or updating), and "io-ratio".
> >
> > The only one so far that I feel is probably better, is
> > "state-update-ratio". If folks feel this one is better than
> > "restore-ratio" I'm happy to update.
> >
> >>
> >> (4) `restore-call-rate`: not sure what you exactly mean by "restore
> calls"?
> >>
> > This is similar to the "io-calls-rate" in the selector classes, i.e.
> > the number of "restore" function calls made. It's argurably a very
> > low-level metrics but I included it since it could be useful in some
> > debugging scenarios.
> >
> >>
> >> (5) `restore-remaining-records-total` -- why is this a task metric?
> >> Seems we could roll it up into a thread metric that we report at INFO
> >> level (we could still have per-task DEBUG level metric for it in
> addition).
> >>
> > The rationale behind it is the general principle in metrics design
> > that "Kafka would provide the lowest necessary metrics levels, and
> > users can do the roll-ups however they want".
> >
> >>
> >> (6) What about "warmup tasks"? Internally, we treat them as standbys,
> >> but it seems it's hard for users to reason about it in the scale-out
> >> warm-up case. Would it be helpful (and possible) to report "warmup
> >> progress" explicitly?
> >>
> > At the restore thread level, we cannot differentiate standby tasks
> > from warmup tasks 

Request for Jira account to contribute to Apache Kafka

[Suprem Vanam]

Dear Apache Software Foundation Team,

I hope this email finds you well. My name is Suprem Vanam, and I am writing
to request a Jira account from ASF. I am a developer and have been learning
Apache Kafka for some time now. Our team has recently started using Kafka
at work, and I have been impressed with the project's capabilities.

I would like to contribute to the Apache Kafka repository on GitHub. I
understand that the infra team has disabled public sign-up due to an influx
of false Jira accounts creating spam tickets. I assure you that I will use
the account responsibly. I understand the importance of maintaining the
integrity of the ASF's systems and will follow all the policies and
guidelines set by the team.

I am excited about the opportunity to contribute to this open-source
project and help make it even better. Please let me know if there is any
additional information required from me.

Thank you for your time and consideration.

Best regards,
Suprem

Re: [DISCUSS] KIP-898: Modernize Connect plugin discovery

[Federico Valeri]

Hi Greg,

On Tue, Jan 24, 2023 at 2:48 AM Greg Harris
 wrote:
>
> Federico,
>
> Thanks for taking a look!
>
> > I was just wondering if we should use a better script name like
> > "connect-convert-to-service-provider.sh" or something like this
>
> I agree that the current name is not ideal, while the current name
> describes part of what it is doing, it does not describe all of what it is
> doing, or why that is important to the caller.
> I looked around to see what style the other Kafka commands use, and it
> seems like there's not a standard pattern. Some of the scripts are
> noun-phrases, and some are verb-phrases.
> Some take commands like `create`, `format`, etc, and some take options like
> `--generate`, `--execute`, `--verify`.
> I've updated the command to `bin/connect-plugin-path.sh
> (list|add-manifests|remove-manifests) --worker-config `, let
> me know if that's better or worse than `bin/connect-scan-plugin-path.sh`.
>
> > maybe add a --dry-run option.
>
> Thanks for the suggestion, I've updated the KIP to include a --dry-run
> option with some typical semantics.

This is much better. Thanks!

I think it's better to not deprecate the add-manifests and
remove-manifests script sub-commands. When we will remove the
deprecated plugin discovery modes, one may still have the need to
convert an old connector release, maybe because it's the only version
compatible with a third-party external system.

>
> Chris,
>
> I'm glad you liked the personas! Since this KIP requires others' help to
> improve the ecosystem, I needed a way to make sure everyone knows what role
> they have to play. I hope I've accomplished that.
>
> > 1. Will workers' resiliency to packaging issues be affected when the new
> > scanning behavior is enabled? For example, will a single poorly-packaged
> > plugin prevent other correctly-packaged plugins from being loaded, or
> > worse, cause the worker to fail? And either way, are there any details
> that
> > we can call out to back up any guarantees we want to provide on that
> front?
>
> The current behavior when encountering packaging issues seems to be to log
> errors and continue, and that will certainly continue for the ONLY_SCAN and
> HYBID_WARN modes.
> As written, the HYBRID_FAIL mode breaks from this trend and will throw
> exceptions that crash the worker only when the connectors are missing from
> serviceloader discovery.
> The behavior for SERVICE_LOAD is not currently defined in the KIP. I think
> we can either keep the log-and-continue behavior whenever possible, or
> begin to surface errors in a fail-fast model.
> And if we can't choose between those two, perhaps we can add
> SERVICE_LOAD_FAIL_FAST, or a plugin.path.discovery.fail.fast=true/false to
> allow users to configure the worker to fail to start up when plugin.path
> issues are present.
>
> With regards to incorrectly packaged connectors affecting other correctly
> packaged ones: I think one of the current behaviors of the Reflections
> implementation is that certain exceptions can prevent discovery of other
> unrelated plugins on the path.
> I think that this may be able to be improved in all modes, and doesn't need
> the ServiceLoader mechanism to resolve. I'll explore this some more and
> pursue a fix outside of this KIP.
> The new mechanism should be as good or better than the Reflections
> mechanism in this regard. I am not sure if it is necessary to commit to the
> error handling behavior as part of the public API.
>
> > 2. I think the description of the "--plugin-location" flag may need some
> > updating? In the middle of the list of bullet points there's "The value of
> > this argument will be a single plugin, which can be any of the following:"
> > but the following points don't appear to be related.
>
> Thanks, I think that was a copy-paste error. I have updated the KIP.
>
> > 3. (Nit) Could we rename the migration script something like
> > "connect-migrate-plugin-path" or even just "connect-plugin-path" with a
> > "migrate" subcommand?
>
> Yes, you and Federico had the same idea :)
> Please see my feedback above regarding naming.
>
> > 4. It's noted that "If a plugin class is removed from the path, the
> > corresponding shim manifest should also be removed". Did you consider
> > adding behavior (possibly guarded behind a CLI flag) to find and remove
> > manifests for nonexistent plugins?
>
> When writing that originally, I thought that the script would only remove
> manifests that it had generated, as filtered by some in-file comment or by
> jar filename, which would be an implementation detail.
> If we were to add a CLI flag to also enable/disable the removal behavior,
> we could eliminate that implementation detail and make the removal an
> all-or-nothing part of the script.
>
> I updated the script to accept a `remove-manifests` subcommand which I
> think captures your suggestion. Therefore, a full sync would include two
> invocations: an `add-manifests` and `remove-manifests`.
>
> > 5. What's the 

[GitHub] [kafka-site] bbejeck merged pull request #464: Powerd by Dream11 section, grammar fixes

[via GitHub]

bbejeck merged PR #464:
URL: https://github.com/apache/kafka-site/pull/464


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] mimaison merged pull request #461: MINOR: Fix docs in security.html

[via GitHub]

mimaison merged PR #461:
URL: https://github.com/apache/kafka-site/pull/461


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [VOTE] KIP-875: First-class offsets support in Kafka Connect

[Knowles Atchison Jr]

+1 (non binding)

On Tue, Jan 24, 2023 at 5:24 AM Yash Mayya  wrote:

> Hi Chris,
>
> I'm +1 (non-binding). Thanks again for proposing this extremely
> valuable addition to Kafka Connect!
>
> Thanks,
> Yash
>
> On Thu, Jan 19, 2023 at 12:11 AM Chris Egerton 
> wrote:
>
> > Hi all,
> >
> > I'd like to call for a vote on KIP-875, which adds support for viewing
> and
> > manipulating the offsets of connectors to the Kafka Connect REST API.
> >
> > The KIP:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect
> >
> > The discussion thread:
> > https://lists.apache.org/thread/m5bklnh5w4mwr9nbzrmfk0pftpxfjd02
> >
> > Cheers,
> >
> > Chris
> >
>

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1537

[Apache Jenkins Server]

See 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.4 #61

[Apache Jenkins Server]

See 

Re: [VOTE] KIP-875: First-class offsets support in Kafka Connect

[Yash Mayya]

Hi Chris,

I'm +1 (non-binding). Thanks again for proposing this extremely
valuable addition to Kafka Connect!

Thanks,
Yash

On Thu, Jan 19, 2023 at 12:11 AM Chris Egerton 
wrote:

> Hi all,
>
> I'd like to call for a vote on KIP-875, which adds support for viewing and
> manipulating the offsets of connectors to the Kafka Connect REST API.
>
> The KIP:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect
>
> The discussion thread:
> https://lists.apache.org/thread/m5bklnh5w4mwr9nbzrmfk0pftpxfjd02
>
> Cheers,
>
> Chris
>

Re: [DISCUSS] KIP-877: Mechanism for plugins and connectors to register metrics

[Yash Mayya]

Hi Mickael,

Thanks for the updated KIP, this is looking really good! I had a couple
more questions -

1) Sensor names need to be unique across all groups for a `Metrics`
instance. How are we planning to avoid naming clashes (both between
different plugins as well as with pre-defined sensors)?

2) Connect has a `ConnectMetrics` wrapper around `Metrics` via which
rebalance / worker / connector / task metrics are recorded. Could you
please elaborate in the KIP how the plugin metrics for connectors / tasks
will inter-operate with this?

Another minor point is that the third rejected alternative appears to be an
incomplete sentence?

Thanks,
Yash

On Fri, Jan 13, 2023 at 10:56 PM Mickael Maison 
wrote:

> Hi,
>
> I've updated the KIP based on the feedback.
>
> Now instead of receiving a Metrics instance, plugins get access to
> PluginMetrics that exposes a much smaller API. I've removed the
> special handling for connectors and tasks and they must now implement
> the Monitorable interface as well to use this feature. Finally the
> goal is to allow all plugins (apart from metrics reporters) to use
> this feature. I've listed them all (there are over 30 pluggable APIs)
> but I've not added the list in the KIP. The reason is that new plugins
> could be added in the future and instead I'll focus on adding support
> in all the place that instantiate classes.
>
> Thanks,
> Mickael
>
> On Tue, Jan 10, 2023 at 7:00 PM Mickael Maison 
> wrote:
> >
> > Hi Chris/Yash,
> >
> > Thanks for taking a look and providing feedback.
> >
> > 1) Yes you're right, when using incompatible version, metrics() would
> > trigger NoSuchMethodError. I thought using the context to pass the
> > Metrics object would be more idiomatic for Connect but maybe
> > implementing Monitorable would be simpler. It would also allow other
> > Connect plugins (transformations, converters, etc) to register
> > metrics. So I'll make that change.
> >
> > 2) As mentioned in the rejected alternatives, I considered having a
> > PluginMetrics class/interface with a limited API. But since Metrics is
> > part of the public API, I thought it would be simpler to reuse it.
> > That said you bring interesting points so I took another look today.
> > It's true that the Metrics API is pretty complex and most methods are
> > useless for plugin authors. I'd expect most use cases only need one
> > addMetric and one sensor methods. Rather than subclassing Metrics, I
> > think a delegate/forwarding pattern might work well here. A
> > PluginMetric class would forward its method to the Metrics instance
> > and could perform some basic validations such as only letting plugins
> > delete metrics they created, or automatically injecting tags with the
> > class name for example.
> >
> > 3) Between the clients, brokers, streams and connect, Kafka has quite
> > a lot! In practice I think registering metrics should be beneficial
> > for all plugins, I think the only exception would be metrics reporters
> > (which are instantiated before the Metrics object). I'll try to build
> > a list of all plugin types and add that to the KIP.
> >
> > Thanks,
> > Mickael
> >
> >
> >
> > On Tue, Dec 27, 2022 at 4:54 PM Chris Egerton 
> wrote:
> > >
> > > Hi Yash,
> > >
> > > Yes, a default no-op is exactly what I had in mind should the
> Connector and
> > > Task classes implement the Monitorable interface.
> > >
> > > Cheers,
> > >
> > > Chris
> > >
> > > On Tue, Dec 20, 2022 at 2:46 AM Yash Mayya 
> wrote:
> > >
> > > > Hi Mickael,
> > > >
> > > > Thanks for creating this KIP, this will be a super useful feature to
> > > > enhance existing connectors in the Kafka Connect ecosystem.
> > > >
> > > > I have some similar concerns to the ones that Chris has outlined
> above,
> > > > especially with regard to directly exposing Connect's Metrics object
> to
> > > > plugins. I believe it would be a lot friendlier to developers if we
> instead
> > > > exposed wrapper methods in the context classes - such as one for
> > > > registering a new metric, one for recording metric values and so on.
> This
> > > > would also have the added benefit of minimizing the surface area for
> > > > potential misuse by custom plugins.
> > > >
> > > > > for connectors and tasks they should handle the
> > > > > metrics() method returning null when deployed on
> > > > > an older runtime.
> > > >
> > > > I believe this won't be the case, and instead they'll need to handle
> a
> > > > `NoSuchMethodError` right? This is similar to previous KIPs that
> added
> > > > methods to connector context classes and will arise due to an
> > > > incompatibility between the `connect-api` dependency that a plugin
> will be
> > > > compiled against versus what it will actually get at runtime.
> > > >
> > > > Hi Chris,
> > > >
> > > > > WDYT about having the Connector and Task classes
> > > > > implement the Monitorable interface, both for
> > > > > consistency's sake, and to prevent classloading
> > > > > headaches?
> > > >
> > > > Are you 

Re: [ANNOUNCE] New committer: Stanislav Kozlovski

[Viktor Somogyi-Vass]

Congrats Stan! :)

On Fri, Jan 20, 2023 at 12:35 AM Colin McCabe  wrote:

> Congratulations, Stan! Well deserved.
>
> best,
> Colin
>
> On Tue, Jan 17, 2023, at 07:50, Jun Rao wrote:
> > Hi, Everyone,
> >
> > The PMC of Apache Kafka is pleased to announce a new Kafka committer
> > Stanislav Kozlovski.
> >
> > Stan has been contributing to Apache Kafka since June 2018. He made
> various
> > contributions including the following KIPs.
> >
> > KIP-455: Create an Administrative API for Replica Reassignment
> > KIP-412: Extend Admin API to support dynamic application log levels
> >
> > Congratulations, Stan!
> >
> > Thanks,
> >
> > Jun (on behalf of the Apache Kafka PMC)
>

Re: [DISCUSS] KIP-877: Mechanism for plugins and connectors to register metrics

[Federico Valeri]

Hi Michael, thanks for the KIP. Looks great.

I noticed that there is a difference on how MetricName and Sensor are
created and removed in PluginMetrics class:

- In metricName() it is not clear what happens if the MetricName
already exists, I guess it will be the same "get or create" behavior
we have on sensor(), but maybe we could clarify in the comment.
- In removeSensor() we do not return any value, so one can't tell if
the Sensor was there or not like with removeMetric().

Minor issue: in the "Example usage" we have setPluginMetrics override,
but should be withPluginMetrics according to the Monitorable
interface.

On Mon, Jan 23, 2023 at 9:01 PM Chris Egerton  wrote:
>
> Hi Mickael,
>
> Thanks for the updates, this is looking great.
>
> I have two more small thoughts:
>
> 1) What's the rationale for defining PluginMetrics as a class instead of an
> interface? AFAICT we don't need a public constructor for it since the
> runtime will be responsible for creating all instances.
>
> 2) The list of affected plugin classes is indeed quite long--thanks for
> listing them all out! I noticed that the ReplicationPolicy interface isn't
> listed for MM2. Is this intentional?
>
> Cheers,
>
> Chris
>
> On Fri, Jan 13, 2023 at 12:26 PM Mickael Maison 
> wrote:
>
> > Hi,
> >
> > I've updated the KIP based on the feedback.
> >
> > Now instead of receiving a Metrics instance, plugins get access to
> > PluginMetrics that exposes a much smaller API. I've removed the
> > special handling for connectors and tasks and they must now implement
> > the Monitorable interface as well to use this feature. Finally the
> > goal is to allow all plugins (apart from metrics reporters) to use
> > this feature. I've listed them all (there are over 30 pluggable APIs)
> > but I've not added the list in the KIP. The reason is that new plugins
> > could be added in the future and instead I'll focus on adding support
> > in all the place that instantiate classes.
> >
> > Thanks,
> > Mickael
> >
> > On Tue, Jan 10, 2023 at 7:00 PM Mickael Maison 
> > wrote:
> > >
> > > Hi Chris/Yash,
> > >
> > > Thanks for taking a look and providing feedback.
> > >
> > > 1) Yes you're right, when using incompatible version, metrics() would
> > > trigger NoSuchMethodError. I thought using the context to pass the
> > > Metrics object would be more idiomatic for Connect but maybe
> > > implementing Monitorable would be simpler. It would also allow other
> > > Connect plugins (transformations, converters, etc) to register
> > > metrics. So I'll make that change.
> > >
> > > 2) As mentioned in the rejected alternatives, I considered having a
> > > PluginMetrics class/interface with a limited API. But since Metrics is
> > > part of the public API, I thought it would be simpler to reuse it.
> > > That said you bring interesting points so I took another look today.
> > > It's true that the Metrics API is pretty complex and most methods are
> > > useless for plugin authors. I'd expect most use cases only need one
> > > addMetric and one sensor methods. Rather than subclassing Metrics, I
> > > think a delegate/forwarding pattern might work well here. A
> > > PluginMetric class would forward its method to the Metrics instance
> > > and could perform some basic validations such as only letting plugins
> > > delete metrics they created, or automatically injecting tags with the
> > > class name for example.
> > >
> > > 3) Between the clients, brokers, streams and connect, Kafka has quite
> > > a lot! In practice I think registering metrics should be beneficial
> > > for all plugins, I think the only exception would be metrics reporters
> > > (which are instantiated before the Metrics object). I'll try to build
> > > a list of all plugin types and add that to the KIP.
> > >
> > > Thanks,
> > > Mickael
> > >
> > >
> > >
> > > On Tue, Dec 27, 2022 at 4:54 PM Chris Egerton 
> > wrote:
> > > >
> > > > Hi Yash,
> > > >
> > > > Yes, a default no-op is exactly what I had in mind should the
> > Connector and
> > > > Task classes implement the Monitorable interface.
> > > >
> > > > Cheers,
> > > >
> > > > Chris
> > > >
> > > > On Tue, Dec 20, 2022 at 2:46 AM Yash Mayya 
> > wrote:
> > > >
> > > > > Hi Mickael,
> > > > >
> > > > > Thanks for creating this KIP, this will be a super useful feature to
> > > > > enhance existing connectors in the Kafka Connect ecosystem.
> > > > >
> > > > > I have some similar concerns to the ones that Chris has outlined
> > above,
> > > > > especially with regard to directly exposing Connect's Metrics object
> > to
> > > > > plugins. I believe it would be a lot friendlier to developers if we
> > instead
> > > > > exposed wrapper methods in the context classes - such as one for
> > > > > registering a new metric, one for recording metric values and so on.
> > This
> > > > > would also have the added benefit of minimizing the surface area for
> > > > > potential misuse by custom plugins.
> > > > >
> > > > > > for connectors and tasks they should handle 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1536

[Apache Jenkins Server]

See