[GitHub] [kafka] erikvanoosten commented on a diff in pull request #13678: KAFKA-10337: await async commits in commitSync even if no offsets given
erikvanoosten commented on code in PR #13678:
URL: https://github.com/apache/kafka/pull/13678#discussion_r1196205881
##
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerCoordinator.java:
##
@@ -1223,6 +1233,26 @@ public void maybeAutoCommitOffsetsAsync(long now) {
}
}
+private boolean waitForPendingAsyncCommits(Timer timer) {
Review Comment:
@philipnee it seems there is still a gap in our mutual understanding. I
really would like this problem to be solved though. It seems to most promising
route towards using Kafka correctly from a completely async/concurrent runtime.
Shall we organize a online meeting, or perhaps you can make the changes you
proposed yourself?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [kafka] erikvanoosten commented on a diff in pull request #13678: KAFKA-10337: await async commits in commitSync even if no offsets given
erikvanoosten commented on code in PR #13678:
URL: https://github.com/apache/kafka/pull/13678#discussion_r1196205881
##
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerCoordinator.java:
##
@@ -1223,6 +1233,26 @@ public void maybeAutoCommitOffsetsAsync(long now) {
}
}
+private boolean waitForPendingAsyncCommits(Timer timer) {
Review Comment:
@philipnee it seems there is still a gap in our mutual understanding. I
really would like this problem to be solved though. It seems the most promising
route towards using Kafka correctly from a completely async/concurrent runtime.
Shall we organize an online meeting, or perhaps you can make the changes you
proposed yourself?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [kafka] jlprat commented on a diff in pull request #13723: KAFKA-14980: Fix MirrorSourceConnector source consumer configuration
jlprat commented on code in PR #13723:
URL: https://github.com/apache/kafka/pull/13723#discussion_r1196222089
##
connect/mirror/src/test/java/org/apache/kafka/connect/mirror/MirrorConnectorConfigTest.java:
##
@@ -43,7 +43,8 @@ protected TestMirrorConnectorConfig(Map
props) {
@Test
public void testSourceConsumerConfig() {
Map connectorProps = makeProps(
-MirrorConnectorConfig.CONSUMER_CLIENT_PREFIX +
"max.poll.interval.ms", "12"
+MirrorConnectorConfig.CONSUMER_CLIENT_PREFIX +
"max.poll.interval.ms", "12",
+MirrorConnectorConfig.SOURCE_CLUSTER_PREFIX +
"bootstrap.servers", "localhost:2345"
Review Comment:
Nice to add a test to avoid the having the same mistake again
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (KAFKA-14999) Kafka Clients not able to publish message when one of broker is shut downed
Sheriff created KAFKA-14999:
---
Summary: Kafka Clients not able to publish message when one of
broker is shut downed
Key: KAFKA-14999
URL: https://issues.apache.org/jira/browse/KAFKA-14999
Project: Kafka
Issue Type: Bug
Components: clients
Affects Versions: 2.6.1
Reporter: Sheriff
Three brokers system on three VMs. One of the VM gracefully shutoff from
openstack GUI. Once VM is shutoff then application is not able to publish
message to Kafka using Java Kafka-clients 2.6.1 getting below execption:
{code:java}
Error sending message: org.apache.kafka.common.errors.TimeoutException: Topic
oes-events not present in metadata after 3 ms. {code}
Even though kafka console producer is able to publish the message.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-14999) Kafka Clients not able to publish message when one of broker is shut downed
[
https://issues.apache.org/jira/browse/KAFKA-14999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sheriff updated KAFKA-14999:
Description:
Three kafka brokers system (three VMs and each VM contains one kafka broker).
One of the VM gracefully shutoff from openstack GUI. Once VM is shutoff then
application is not able to publish message to Kafka using Java Kafka-clients
2.6.1 getting below execption:
{code:java}
Error sending message: org.apache.kafka.common.errors.TimeoutException: Topic
oes-events not present in metadata after 3 ms. {code}
Even though kafka console producer is able to publish the message.
was:
Three brokers system on three VMs. One of the VM gracefully shutoff from
openstack GUI. Once VM is shutoff then application is not able to publish
message to Kafka using Java Kafka-clients 2.6.1 getting below execption:
{code:java}
Error sending message: org.apache.kafka.common.errors.TimeoutException: Topic
oes-events not present in metadata after 3 ms. {code}
Even though kafka console producer is able to publish the message.
> Kafka Clients not able to publish message when one of broker is shut downed
>
>
> Key: KAFKA-14999
> URL: https://issues.apache.org/jira/browse/KAFKA-14999
> Project: Kafka
> Issue Type: Bug
> Components: clients
>Affects Versions: 2.6.1
>Reporter: Sheriff
>Priority: Major
>
> Three kafka brokers system (three VMs and each VM contains one kafka
> broker). One of the VM gracefully shutoff from openstack GUI. Once VM is
> shutoff then application is not able to publish message to Kafka using Java
> Kafka-clients 2.6.1 getting below execption:
>
> {code:java}
> Error sending message: org.apache.kafka.common.errors.TimeoutException: Topic
> oes-events not present in metadata after 3 ms. {code}
>
> Even though kafka console producer is able to publish the message.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [kafka] satishd commented on pull request #13535: KAFKA-9579 Fetch implementation for records in the remote storage through a specific purgatory.
satishd commented on PR #13535: URL: https://github.com/apache/kafka/pull/13535#issuecomment-1551099626 Thanks @junrao for the updated review. Addressed your latest minor review comments. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (KAFKA-14999) Kafka Clients not able to publish message when one of broker is shut downed
[
https://issues.apache.org/jira/browse/KAFKA-14999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sheriff updated KAFKA-14999:
Description:
Three kafka brokers system (three VMs and each VM contains one kafka broker).
One of the VM gracefully shutoff from openstack GUI. Once VM is shutoff then
application is not able to publish message to Kafka using Java Kafka-clients
2.6.1 getting below execption:
{code:java}
Error sending message: org.apache.kafka.common.errors.TimeoutException: Topic
test-events not present in metadata after 3 ms. {code}
Even though kafka console producer is able to publish the message.
was:
Three kafka brokers system (three VMs and each VM contains one kafka broker).
One of the VM gracefully shutoff from openstack GUI. Once VM is shutoff then
application is not able to publish message to Kafka using Java Kafka-clients
2.6.1 getting below execption:
{code:java}
Error sending message: org.apache.kafka.common.errors.TimeoutException: Topic
oes-events not present in metadata after 3 ms. {code}
Even though kafka console producer is able to publish the message.
> Kafka Clients not able to publish message when one of broker is shut downed
>
>
> Key: KAFKA-14999
> URL: https://issues.apache.org/jira/browse/KAFKA-14999
> Project: Kafka
> Issue Type: Bug
> Components: clients
>Affects Versions: 2.6.1
>Reporter: Sheriff
>Priority: Major
>
> Three kafka brokers system (three VMs and each VM contains one kafka
> broker). One of the VM gracefully shutoff from openstack GUI. Once VM is
> shutoff then application is not able to publish message to Kafka using Java
> Kafka-clients 2.6.1 getting below execption:
>
> {code:java}
> Error sending message: org.apache.kafka.common.errors.TimeoutException: Topic
> test-events not present in metadata after 3 ms. {code}
>
> Even though kafka console producer is able to publish the message.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [kafka] hudeqi commented on pull request #13719: MINOR:Fix illogical log in fetchOffsetAndTruncate method
hudeqi commented on PR #13719: URL: https://github.com/apache/kafka/pull/13719#issuecomment-1551304423 Hi, do you have time to help review this? @dajac @satishd -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] vamossagar12 opened a new pull request, #13726: KAFKA-13109: SourceTask#poll not enforcing retries in case of RetriableException
vamossagar12 opened a new pull request, #13726: URL: https://github.com/apache/kafka/pull/13726 AbstractWorkerSourceTask is not enforcing the errors.retry.timeout and errors.retry.delay.max.ms parameters in case of a RetriableException during task.poll(). [KIP-298 ](https://cwiki.apache.org/confluence/display/KAFKA/KIP-298%3A+Error+Handling+in+Connect) explicitly mentions that `poll() in SourceTask will retry in case of RetriableException. The current behaviour is that in case of RetriableException, null is returned immediately. This PR aims to add that support. Note that I think SinkTask#put doesn't honour which can be taken up in a follow up PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] hudeqi commented on pull request #13696: KAFKA-14979:Incorrect lag was calculated when markPartitionsForTruncation in ReplicaAlterLogDirsThread
hudeqi commented on PR #13696: URL: https://github.com/apache/kafka/pull/13696#issuecomment-1551312847 hi, do you have time to review this issue？ @showuon -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (KAFKA-12384) Flaky Test ListOffsetsRequestTest.testResponseIncludesLeaderEpoch
[
https://issues.apache.org/jira/browse/KAFKA-12384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17723454#comment-17723454
]
Satish Duggana commented on KAFKA-12384:
Reopening as these tests are still flaky and failing in the latest CI job runs.
https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-13535/32/tests/
> Flaky Test ListOffsetsRequestTest.testResponseIncludesLeaderEpoch
> -
>
> Key: KAFKA-12384
> URL: https://issues.apache.org/jira/browse/KAFKA-12384
> Project: Kafka
> Issue Type: Test
> Components: core, unit tests
>Reporter: Matthias J. Sax
>Assignee: Chia-Ping Tsai
>Priority: Critical
> Labels: flaky-test
> Fix For: 3.0.0
>
>
> {quote}org.opentest4j.AssertionFailedError: expected: <(0,0)> but was:
> <(-1,-1)> at
> org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:55) at
> org.junit.jupiter.api.AssertionUtils.failNotEqual(AssertionUtils.java:62) at
> org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182) at
> org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:177) at
> org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:1124) at
> kafka.server.ListOffsetsRequestTest.testResponseIncludesLeaderEpoch(ListOffsetsRequestTest.scala:172){quote}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Reopened] (KAFKA-12384) Flaky Test ListOffsetsRequestTest.testResponseIncludesLeaderEpoch
[
https://issues.apache.org/jira/browse/KAFKA-12384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Satish Duggana reopened KAFKA-12384:
> Flaky Test ListOffsetsRequestTest.testResponseIncludesLeaderEpoch
> -
>
> Key: KAFKA-12384
> URL: https://issues.apache.org/jira/browse/KAFKA-12384
> Project: Kafka
> Issue Type: Test
> Components: core, unit tests
>Reporter: Matthias J. Sax
>Assignee: Chia-Ping Tsai
>Priority: Critical
> Labels: flaky-test
> Fix For: 3.0.0
>
>
> {quote}org.opentest4j.AssertionFailedError: expected: <(0,0)> but was:
> <(-1,-1)> at
> org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:55) at
> org.junit.jupiter.api.AssertionUtils.failNotEqual(AssertionUtils.java:62) at
> org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182) at
> org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:177) at
> org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:1124) at
> kafka.server.ListOffsetsRequestTest.testResponseIncludesLeaderEpoch(ListOffsetsRequestTest.scala:172){quote}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [kafka] satishd merged pull request #13718: MINOR: move ControlRecordTest to correct directory
satishd merged PR #13718: URL: https://github.com/apache/kafka/pull/13718 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] dengziming commented on a diff in pull request #13562: KAFKA-14581: Moving GetOffsetShell to tools
dengziming commented on code in PR #13562:
URL: https://github.com/apache/kafka/pull/13562#discussion_r1196474288
##
tools/src/test/java/org/apache/kafka/tools/GetOffsetShellTest.java:
##
@@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.tools;
+
+import kafka.test.ClusterInstance;
+import kafka.test.annotation.ClusterTest;
+import kafka.test.annotation.ClusterTestDefaults;
+import kafka.test.annotation.Type;
+import kafka.test.junit.ClusterTestExtensions;
+import org.apache.kafka.clients.CommonClientConfigs;
+import org.apache.kafka.clients.admin.Admin;
+import org.apache.kafka.clients.admin.NewTopic;
+import org.apache.kafka.clients.producer.KafkaProducer;
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.extension.ExtendWith;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import java.util.stream.Stream;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+@ExtendWith(value = ClusterTestExtensions.class)
+@ClusterTestDefaults(clusterType = Type.ZK)
+@Tag("integration")
+public class GetOffsetShellTest {
+private final int topicCount = 4;
+private final int offsetTopicPartitionCount = 4;
+private final ClusterInstance cluster;
+
+public GetOffsetShellTest(ClusterInstance cluster) {
+this.cluster = cluster;
+}
+
+private String getTopicName(int i) {
+return "topic" + i;
+}
+
+@BeforeEach
+public void setUp() {
Review Comment:
we should pass a `clusterConfig: ClusterConfig` parameter to setUp.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (KAFKA-15000) High vulnerability PRISMA-2023-0067 reported in jackson-core
Arushi Rai created KAFKA-15000: -- Summary: High vulnerability PRISMA-2023-0067 reported in jackson-core Key: KAFKA-15000 URL: https://issues.apache.org/jira/browse/KAFKA-15000 Project: Kafka Issue Type: Bug Affects Versions: 3.3.2, 3.4.0 Reporter: Arushi Rai Kafka is using jackson-core version 2.13.4 which has high vulnerability reported [PRISMA-2023-0067. |https://github.com/FasterXML/jackson-core/pull/827] This vulnerability is fix in Jackson-core 2.15.0 and Kafka should upgrade to the same. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-15000) High vulnerability PRISMA-2023-0067 reported in jackson-core
[ https://issues.apache.org/jira/browse/KAFKA-15000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17723486#comment-17723486 ] Manyanda Chitimbo commented on KAFKA-15000: --- The PR [https://github.com/apache/kafka/pull/13662] should address the issue. Maybe when it is merged it can be backported. > High vulnerability PRISMA-2023-0067 reported in jackson-core > > > Key: KAFKA-15000 > URL: https://issues.apache.org/jira/browse/KAFKA-15000 > Project: Kafka > Issue Type: Bug >Affects Versions: 3.4.0, 3.3.2 >Reporter: Arushi Rai >Priority: Critical > > Kafka is using jackson-core version 2.13.4 which has high vulnerability > reported [PRISMA-2023-0067. > |https://github.com/FasterXML/jackson-core/pull/827] > This vulnerability is fix in Jackson-core 2.15.0 and Kafka should upgrade to > the same. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-15000) High vulnerability PRISMA-2023-0067 reported in jackson-core
[ https://issues.apache.org/jira/browse/KAFKA-15000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17723489#comment-17723489 ] Arushi Rai commented on KAFKA-15000: Hi [~manyanda] Thank you for the response. I do not see milestone added in the ticket but do you know what is the expected release version for this? > High vulnerability PRISMA-2023-0067 reported in jackson-core > > > Key: KAFKA-15000 > URL: https://issues.apache.org/jira/browse/KAFKA-15000 > Project: Kafka > Issue Type: Bug >Affects Versions: 3.4.0, 3.3.2 >Reporter: Arushi Rai >Priority: Critical > > Kafka is using jackson-core version 2.13.4 which has high vulnerability > reported [PRISMA-2023-0067. > |https://github.com/FasterXML/jackson-core/pull/827] > This vulnerability is fix in Jackson-core 2.15.0 and Kafka should upgrade to > the same. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-15000) High vulnerability PRISMA-2023-0067 reported in jackson-core
[ https://issues.apache.org/jira/browse/KAFKA-15000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17723502#comment-17723502 ] Manyanda Chitimbo commented on KAFKA-15000: --- Hi [~arushir] I've no idea but let me tag along [~showuon] [~mimaison] who I am hoping will have a better idea than myself. > High vulnerability PRISMA-2023-0067 reported in jackson-core > > > Key: KAFKA-15000 > URL: https://issues.apache.org/jira/browse/KAFKA-15000 > Project: Kafka > Issue Type: Bug >Affects Versions: 3.4.0, 3.3.2 >Reporter: Arushi Rai >Priority: Critical > > Kafka is using jackson-core version 2.13.4 which has high vulnerability > reported [PRISMA-2023-0067. > |https://github.com/FasterXML/jackson-core/pull/827] > This vulnerability is fix in Jackson-core 2.15.0 and Kafka should upgrade to > the same. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-15001) Medium vulnerabilities in Jetty
Arushi Rai created KAFKA-15001: -- Summary: Medium vulnerabilities in Jetty Key: KAFKA-15001 URL: https://issues.apache.org/jira/browse/KAFKA-15001 Project: Kafka Issue Type: Task Affects Versions: 3.3.2, 3.4.0 Reporter: Arushi Rai Kafka is using org.eclipse.jetty_jetty-server and org.eclipse.jetty_jetty-io version 9.4.48.v20220622 where 3 moderate and medium vulnerabilities have been reported. Moderate [CVE-2023-26048|https://nvd.nist.gov/vuln/detail/CVE-2023-26048] in org.eclipse.jetty_jetty-server Medium [CVE-2023-26049|https://nvd.nist.gov/vuln/detail/CVE-2023-26049] in org.eclipse.jetty_jetty-io Medium [CVE-2023-26048|https://nvd.nist.gov/vuln/detail/CVE-2023-26048] in org.eclipse.jetty_jetty-io These are fixed in jetty versions 11.0.14, 10.0.14, 9.4.51 and Kafka should use the same. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-15002) Moderate vulnerability in org.bitbucket.b_c_jose4j
Arushi Rai created KAFKA-15002: -- Summary: Moderate vulnerability in org.bitbucket.b_c_jose4j Key: KAFKA-15002 URL: https://issues.apache.org/jira/browse/KAFKA-15002 Project: Kafka Issue Type: Task Affects Versions: 3.3.2, 3.4.0 Reporter: Arushi Rai Kafka is using package org.bitbucket.b_c_jose4j on version 0.7.9 where medium vulnerability is reported [GHSA-jgvc-jfgh-rjvv|https://github.com/advisories/GHSA-jgvc-jfgh-rjvv]. Fix is available in version 0.9.3 and Kafka should look to update to the fix version. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-15003) TopicIdReplicaAssignment is not updated in migration (dual-write) when partitions are changed for topic
Akhilesh Chaganti created KAFKA-15003: - Summary: TopicIdReplicaAssignment is not updated in migration (dual-write) when partitions are changed for topic Key: KAFKA-15003 URL: https://issues.apache.org/jira/browse/KAFKA-15003 Project: Kafka Issue Type: Bug Components: kraft Affects Versions: 3.5.0 Reporter: Akhilesh Chaganti Fix For: 3.5.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-15004) Topic config changes are not synced during zk to kraft migration (dual-write)
Akhilesh Chaganti created KAFKA-15004: - Summary: Topic config changes are not synced during zk to kraft migration (dual-write) Key: KAFKA-15004 URL: https://issues.apache.org/jira/browse/KAFKA-15004 Project: Kafka Issue Type: Bug Reporter: Akhilesh Chaganti -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [kafka] junrao commented on a diff in pull request #13535: KAFKA-9579 Fetch implementation for records in the remote storage through a specific purgatory.
junrao commented on code in PR #13535:
URL: https://github.com/apache/kafka/pull/13535#discussion_r1196786120
##
core/src/main/scala/kafka/server/ReplicaManager.scala:
##
@@ -1175,49 +1260,78 @@ class ReplicaManager(val config: KafkaConfig,
fetchPartitionStatus += (topicIdPartition ->
FetchPartitionStatus(logOffsetMetadata, partitionData))
})
}
- val delayedFetch = new DelayedFetch(
-params = params,
-fetchPartitionStatus = fetchPartitionStatus,
-replicaManager = this,
-quota = quota,
-responseCallback = responseCallback
- )
-
- // create a list of (topic, partition) pairs to use as keys for this
delayed fetch operation
- val delayedFetchKeys = fetchPartitionStatus.map { case (tp, _) =>
TopicPartitionOperationKey(tp) }
-
- // try to complete the request immediately, otherwise put it into the
purgatory;
- // this is because while the delayed fetch operation is being created,
new requests
- // may arrive and hence make this operation completable.
- delayedFetchPurgatory.tryCompleteElseWatch(delayedFetch,
delayedFetchKeys)
+
+ if (remoteFetchInfo.isPresent) {
+val maybeLogReadResultWithError =
processRemoteFetch(remoteFetchInfo.get(), params, responseCallback,
logReadResults, fetchPartitionStatus)
+if(maybeLogReadResultWithError.isDefined) {
Review Comment:
space after `if `
##
core/src/main/scala/kafka/server/ReplicaManager.scala:
##
@@ -1175,49 +1260,78 @@ class ReplicaManager(val config: KafkaConfig,
fetchPartitionStatus += (topicIdPartition ->
FetchPartitionStatus(logOffsetMetadata, partitionData))
})
}
- val delayedFetch = new DelayedFetch(
-params = params,
-fetchPartitionStatus = fetchPartitionStatus,
-replicaManager = this,
-quota = quota,
-responseCallback = responseCallback
- )
-
- // create a list of (topic, partition) pairs to use as keys for this
delayed fetch operation
- val delayedFetchKeys = fetchPartitionStatus.map { case (tp, _) =>
TopicPartitionOperationKey(tp) }
-
- // try to complete the request immediately, otherwise put it into the
purgatory;
- // this is because while the delayed fetch operation is being created,
new requests
- // may arrive and hence make this operation completable.
- delayedFetchPurgatory.tryCompleteElseWatch(delayedFetch,
delayedFetchKeys)
+
+ if (remoteFetchInfo.isPresent) {
+val maybeLogReadResultWithError =
processRemoteFetch(remoteFetchInfo.get(), params, responseCallback,
logReadResults, fetchPartitionStatus)
+if(maybeLogReadResultWithError.isDefined) {
+ // If there is an error in scheduling the remote fetch task, return
what we currently have
+ // (the data read from local log segment for the other
topic-partitions) and an error for the topic-partition
+ // that we couldn't read from remote storage
+ val partitionToFetchPartitionData =
buildPartitionToFetchPartitionData(logReadResults,
remoteFetchInfo.get().topicPartition, maybeLogReadResultWithError.get)
+ responseCallback(partitionToFetchPartitionData)
+ return
Review Comment:
It seems that `return` is unnecessary here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [kafka] splett2 opened a new pull request, #13727: MINOR: Rename handleSnapshot to handleLoadSnapshot
splett2 opened a new pull request, #13727: URL: https://github.com/apache/kafka/pull/13727 This PR tries to make it explicit that `handleSnapshot` is handling the loading of a snapshot. To someone who is not familiar with the code or KRaft's design, `handleSnapshot` may sound like the event is a snapshot event, whereas it is really a snapshot loading event. ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (KAFKA-15005) Status of KafkaConnect task not correct
Yu Wang created KAFKA-15005:
---
Summary: Status of KafkaConnect task not correct
Key: KAFKA-15005
URL: https://issues.apache.org/jira/browse/KAFKA-15005
Project: Kafka
Issue Type: Bug
Components: KafkaConnect
Affects Versions: 3.3.2, 3.0.0, 2.5.1
Reporter: Yu Wang
A rebalance of our MM2 source tasks, we found there are several tasks always in
*UNASSIGNED* status.
So we dump the payload of the status topic of Kafka Connect, and found the last
two status change is status *RUNNING* followed by status {*}UNASSIGNED{*}.
{code:java}
LogAppendTime: 1684167885961 keysize: 64 valuesize: 95 sequence: -1 headerKeys:
[] key: task-7 payload:
{"state":"RUNNING","trace":null,"worker_id":"x","generation":437643}
LogAppendTime: 1684167903456 keysize: 64 valuesize: 98 sequence: -1 headerKeys:
[] key: task-7 payload:
{"state":"UNASSIGNED","trace":null,"worker_id":"x","generation":437643}
{code}
Usually the RUNNING status should be appended after the UNASSIGNED, because the
worker coordinator will revoked the tasks before start new tasks.
Then we checked the log of our MM2 worker. And found that, during that time,
there was a task that revoked on worker-2 and started on worker-1.
Worker-1
{code:java}
[2023-05-15 09:24:45,950] INFO [Worker clientId=connect-1, groupId=__group]
Starting task task-7
(org.apache.kafka.connect.runtime.distributed.DistributedHerder)
[2023-05-15 09:24:45,951] INFO Creating task task-7
(org.apache.kafka.connect.runtime.Worker) {code}
Worker-2
{code:java}
[2023-05-15 09:24:40,922] INFO Stopping task task-7
(org.apache.kafka.connect.runtime.Worker) {code}
So I think the incorrect status was caused by the revoked task finished later
than the new started task, which made the UNASSIGNED status append to that
status topic after the RUNNING status.
After reading the code of DistributeHerder, I found that the task revoking is
running in a thread pool, the revoke operation just return after submit all the
callables. So I think even in the same worker, there is not a guarantee that
the revoke operation will always finish before the new tasks start.
{code:java}
for (final ConnectorTaskId taskId : tasks) {
callables.add(getTaskStoppingCallable(taskId));
}
// The actual timeout for graceful task/connector stop is applied in worker's
// stopAndAwaitTask/stopAndAwaitConnector methods.
startAndStop(callables);
log.info("Finished stopping tasks in preparation for rebalance"); {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-15005) Status of KafkaConnect task not correct
[
https://issues.apache.org/jira/browse/KAFKA-15005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Wang updated KAFKA-15005:
Description:
After a rebalance of our MM2 source tasks, we found there were several tasks
always in *UNASSIGNED* status, event the real tasks already started.
So we dump the payload of the status topic of Kafka Connect, and found the last
two status change is status *RUNNING* followed by status {*}UNASSIGNED{*}.
{code:java}
LogAppendTime: 1684167885961 keysize: 64 valuesize: 95 sequence: -1 headerKeys:
[] key: task-7 payload:
{"state":"RUNNING","trace":null,"worker_id":"x","generation":437643}
LogAppendTime: 1684167903456 keysize: 64 valuesize: 98 sequence: -1 headerKeys:
[] key: task-7 payload:
{"state":"UNASSIGNED","trace":null,"worker_id":"x","generation":437643}
{code}
Usually the RUNNING status should be appended after the UNASSIGNED, because the
worker coordinator will revoked the tasks before start new tasks.
Then we checked the log of our MM2 worker. And found that, during that time,
there was a task that revoked on worker-2 and started on worker-1.
Worker-1
{code:java}
[2023-05-15 09:24:45,950] INFO [Worker clientId=connect-1, groupId=__group]
Starting task task-7
(org.apache.kafka.connect.runtime.distributed.DistributedHerder)
[2023-05-15 09:24:45,951] INFO Creating task task-7
(org.apache.kafka.connect.runtime.Worker) {code}
Worker-2
{code:java}
[2023-05-15 09:24:40,922] INFO Stopping task task-7
(org.apache.kafka.connect.runtime.Worker) {code}
So I think the incorrect status was caused by the revoked task finished later
than the new started task, which made the UNASSIGNED status append to that
status topic after the RUNNING status.
After reading the code of DistributeHerder, I found that the task revoking is
running in a thread pool, the revoke operation just return after submit all the
callables. So I think even in the same worker, there is not a guarantee that
the revoke operation will always finish before the new tasks start.
{code:java}
for (final ConnectorTaskId taskId : tasks) {
callables.add(getTaskStoppingCallable(taskId));
}
// The actual timeout for graceful task/connector stop is applied in worker's
// stopAndAwaitTask/stopAndAwaitConnector methods.
startAndStop(callables);
log.info("Finished stopping tasks in preparation for rebalance"); {code}
was:
A rebalance of our MM2 source tasks, we found there are several tasks always in
*UNASSIGNED* status.
So we dump the payload of the status topic of Kafka Connect, and found the last
two status change is status *RUNNING* followed by status {*}UNASSIGNED{*}.
{code:java}
LogAppendTime: 1684167885961 keysize: 64 valuesize: 95 sequence: -1 headerKeys:
[] key: task-7 payload:
{"state":"RUNNING","trace":null,"worker_id":"x","generation":437643}
LogAppendTime: 1684167903456 keysize: 64 valuesize: 98 sequence: -1 headerKeys:
[] key: task-7 payload:
{"state":"UNASSIGNED","trace":null,"worker_id":"x","generation":437643}
{code}
Usually the RUNNING status should be appended after the UNASSIGNED, because the
worker coordinator will revoked the tasks before start new tasks.
Then we checked the log of our MM2 worker. And found that, during that time,
there was a task that revoked on worker-2 and started on worker-1.
Worker-1
{code:java}
[2023-05-15 09:24:45,950] INFO [Worker clientId=connect-1, groupId=__group]
Starting task task-7
(org.apache.kafka.connect.runtime.distributed.DistributedHerder)
[2023-05-15 09:24:45,951] INFO Creating task task-7
(org.apache.kafka.connect.runtime.Worker) {code}
Worker-2
{code:java}
[2023-05-15 09:24:40,922] INFO Stopping task task-7
(org.apache.kafka.connect.runtime.Worker) {code}
So I think the incorrect status was caused by the revoked task finished later
than the new started task, which made the UNASSIGNED status append to that
status topic after the RUNNING status.
After reading the code of DistributeHerder, I found that the task revoking is
running in a thread pool, the revoke operation just return after submit all the
callables. So I think even in the same worker, there is not a guarantee that
the revoke operation will always finish before the new tasks start.
{code:java}
for (final ConnectorTaskId taskId : tasks) {
callables.add(getTaskStoppingCallable(taskId));
}
// The actual timeout for graceful task/connector stop is applied in worker's
// stopAndAwaitTask/stopAndAwaitConnector methods.
startAndStop(callables);
log.info("Finished stopping tasks in preparation for rebalance"); {code}
> Status of KafkaConnect task not correct
> ---
>
> Key: KAFKA-15005
> URL: https://issues.apache.org/jira/browse/KAFKA-15005
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Affects Versions: 2.5.1, 3.0.0, 3.3.2
>
[jira] [Updated] (KAFKA-15005) Status of KafkaConnect task not correct
[
https://issues.apache.org/jira/browse/KAFKA-15005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Wang updated KAFKA-15005:
Description:
After a rebalance of our MM2 source tasks, we found there were several tasks
always in *UNASSIGNED* status, event the real tasks already started.
So we dump the payload of the status topic of Kafka Connect, and found the last
two status change is status *RUNNING* followed by status {*}UNASSIGNED{*}.
{code:java}
LogAppendTime: 1684167885961 keysize: 64 valuesize: 95 sequence: -1 headerKeys:
[] key: task-7 payload:
{"state":"RUNNING","trace":null,"worker_id":"x","generation":437643}
LogAppendTime: 1684167903456 keysize: 64 valuesize: 98 sequence: -1 headerKeys:
[] key: task-7 payload:
{"state":"UNASSIGNED","trace":null,"worker_id":"x","generation":437643}
{code}
But usually, the RUNNING status should be appended after the UNASSIGNED,
because the worker coordinator will revoked the tasks before start new tasks.
Then we checked the log of our MM2 worker. And found that, during that time,
there was a task that revoked on worker-2 and started on worker-1.
Worker-1
{code:java}
[2023-05-15 09:24:45,950] INFO [Worker clientId=connect-1, groupId=__group]
Starting task task-7
(org.apache.kafka.connect.runtime.distributed.DistributedHerder)
[2023-05-15 09:24:45,951] INFO Creating task task-7
(org.apache.kafka.connect.runtime.Worker) {code}
Worker-2
{code:java}
[2023-05-15 09:24:40,922] INFO Stopping task task-7
(org.apache.kafka.connect.runtime.Worker) {code}
So I think the incorrect status was caused by the revoked task finished later
than the new started task, which made the UNASSIGNED status append to that
status topic after the RUNNING status.
After reading the code of DistributeHerder, I found that the task revoking is
running in a thread pool, the revoke operation just return after submit all the
callables. So I think even in the same worker, there is not a guarantee that
the revoke operation will always finish before the new tasks start.
{code:java}
for (final ConnectorTaskId taskId : tasks) {
callables.add(getTaskStoppingCallable(taskId));
}
// The actual timeout for graceful task/connector stop is applied in worker's
// stopAndAwaitTask/stopAndAwaitConnector methods.
startAndStop(callables);
log.info("Finished stopping tasks in preparation for rebalance"); {code}
was:
After a rebalance of our MM2 source tasks, we found there were several tasks
always in *UNASSIGNED* status, event the real tasks already started.
So we dump the payload of the status topic of Kafka Connect, and found the last
two status change is status *RUNNING* followed by status {*}UNASSIGNED{*}.
{code:java}
LogAppendTime: 1684167885961 keysize: 64 valuesize: 95 sequence: -1 headerKeys:
[] key: task-7 payload:
{"state":"RUNNING","trace":null,"worker_id":"x","generation":437643}
LogAppendTime: 1684167903456 keysize: 64 valuesize: 98 sequence: -1 headerKeys:
[] key: task-7 payload:
{"state":"UNASSIGNED","trace":null,"worker_id":"x","generation":437643}
{code}
Usually the RUNNING status should be appended after the UNASSIGNED, because the
worker coordinator will revoked the tasks before start new tasks.
Then we checked the log of our MM2 worker. And found that, during that time,
there was a task that revoked on worker-2 and started on worker-1.
Worker-1
{code:java}
[2023-05-15 09:24:45,950] INFO [Worker clientId=connect-1, groupId=__group]
Starting task task-7
(org.apache.kafka.connect.runtime.distributed.DistributedHerder)
[2023-05-15 09:24:45,951] INFO Creating task task-7
(org.apache.kafka.connect.runtime.Worker) {code}
Worker-2
{code:java}
[2023-05-15 09:24:40,922] INFO Stopping task task-7
(org.apache.kafka.connect.runtime.Worker) {code}
So I think the incorrect status was caused by the revoked task finished later
than the new started task, which made the UNASSIGNED status append to that
status topic after the RUNNING status.
After reading the code of DistributeHerder, I found that the task revoking is
running in a thread pool, the revoke operation just return after submit all the
callables. So I think even in the same worker, there is not a guarantee that
the revoke operation will always finish before the new tasks start.
{code:java}
for (final ConnectorTaskId taskId : tasks) {
callables.add(getTaskStoppingCallable(taskId));
}
// The actual timeout for graceful task/connector stop is applied in worker's
// stopAndAwaitTask/stopAndAwaitConnector methods.
startAndStop(callables);
log.info("Finished stopping tasks in preparation for rebalance"); {code}
> Status of KafkaConnect task not correct
> ---
>
> Key: KAFKA-15005
> URL: https://issues.apache.org/jira/browse/KAFKA-15005
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConn
[jira] [Updated] (KAFKA-15005) Status of KafkaConnect task not correct
[
https://issues.apache.org/jira/browse/KAFKA-15005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Wang updated KAFKA-15005:
Description:
Our MM2 is running version 2.5.1.
After a rebalance of our MM2 source tasks, we found there were several tasks
always in *UNASSIGNED* status, even the real tasks already started.
So we dump the payload of the status topic of Kafka Connect, and found the last
two status change is status *RUNNING* followed by status {*}UNASSIGNED{*}.
{code:java}
LogAppendTime: 1684167885961 keysize: 64 valuesize: 95 sequence: -1 headerKeys:
[] key: task-7 payload:
{"state":"RUNNING","trace":null,"worker_id":"x","generation":437643}
LogAppendTime: 1684167903456 keysize: 64 valuesize: 98 sequence: -1 headerKeys:
[] key: task-7 payload:
{"state":"UNASSIGNED","trace":null,"worker_id":"x","generation":437643}
{code}
But usually, the RUNNING status should be appended after the UNASSIGNED,
because the worker coordinator will revoked the tasks before start new tasks.
Then we checked the log of our MM2 worker. And found that, during that time,
there was a task that revoked on worker-2 and started on worker-1.
Worker-1
{code:java}
[2023-05-15 09:24:45,950] INFO [Worker clientId=connect-1, groupId=__group]
Starting task task-7
(org.apache.kafka.connect.runtime.distributed.DistributedHerder)
[2023-05-15 09:24:45,951] INFO Creating task task-7
(org.apache.kafka.connect.runtime.Worker) {code}
Worker-2
{code:java}
[2023-05-15 09:24:40,922] INFO Stopping task task-7
(org.apache.kafka.connect.runtime.Worker) {code}
So I think the incorrect status was caused by the revoked task finished later
than the new started task, which made the UNASSIGNED status append to that
status topic after the RUNNING status.
After reading the code of DistributeHerder, I found that the task revoking is
running in a thread pool, the revoke operation just return after submit all the
callables. So I think even in the same worker, there is not a guarantee that
the revoke operation will always finish before the new tasks start.
{code:java}
for (final ConnectorTaskId taskId : tasks) {
callables.add(getTaskStoppingCallable(taskId));
}
// The actual timeout for graceful task/connector stop is applied in worker's
// stopAndAwaitTask/stopAndAwaitConnector methods.
startAndStop(callables);
log.info("Finished stopping tasks in preparation for rebalance"); {code}
was:
Our MM2 is running version 2.5.1.
After a rebalance of our MM2 source tasks, we found there were several tasks
always in *UNASSIGNED* status, event the real tasks already started.
So we dump the payload of the status topic of Kafka Connect, and found the last
two status change is status *RUNNING* followed by status {*}UNASSIGNED{*}.
{code:java}
LogAppendTime: 1684167885961 keysize: 64 valuesize: 95 sequence: -1 headerKeys:
[] key: task-7 payload:
{"state":"RUNNING","trace":null,"worker_id":"x","generation":437643}
LogAppendTime: 1684167903456 keysize: 64 valuesize: 98 sequence: -1 headerKeys:
[] key: task-7 payload:
{"state":"UNASSIGNED","trace":null,"worker_id":"x","generation":437643}
{code}
But usually, the RUNNING status should be appended after the UNASSIGNED,
because the worker coordinator will revoked the tasks before start new tasks.
Then we checked the log of our MM2 worker. And found that, during that time,
there was a task that revoked on worker-2 and started on worker-1.
Worker-1
{code:java}
[2023-05-15 09:24:45,950] INFO [Worker clientId=connect-1, groupId=__group]
Starting task task-7
(org.apache.kafka.connect.runtime.distributed.DistributedHerder)
[2023-05-15 09:24:45,951] INFO Creating task task-7
(org.apache.kafka.connect.runtime.Worker) {code}
Worker-2
{code:java}
[2023-05-15 09:24:40,922] INFO Stopping task task-7
(org.apache.kafka.connect.runtime.Worker) {code}
So I think the incorrect status was caused by the revoked task finished later
than the new started task, which made the UNASSIGNED status append to that
status topic after the RUNNING status.
After reading the code of DistributeHerder, I found that the task revoking is
running in a thread pool, the revoke operation just return after submit all the
callables. So I think even in the same worker, there is not a guarantee that
the revoke operation will always finish before the new tasks start.
{code:java}
for (final ConnectorTaskId taskId : tasks) {
callables.add(getTaskStoppingCallable(taskId));
}
// The actual timeout for graceful task/connector stop is applied in worker's
// stopAndAwaitTask/stopAndAwaitConnector methods.
startAndStop(callables);
log.info("Finished stopping tasks in preparation for rebalance"); {code}
> Status of KafkaConnect task not correct
> ---
>
> Key: KAFKA-15005
> URL: https://issues.apache.org/jira/browse/KAFKA-15005
>
[jira] [Updated] (KAFKA-15005) Status of KafkaConnect task not correct
[
https://issues.apache.org/jira/browse/KAFKA-15005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Wang updated KAFKA-15005:
Description:
Our MM2 is running version 2.5.1.
After a rebalance of our MM2 source tasks, we found there were several tasks
always in *UNASSIGNED* status, event the real tasks already started.
So we dump the payload of the status topic of Kafka Connect, and found the last
two status change is status *RUNNING* followed by status {*}UNASSIGNED{*}.
{code:java}
LogAppendTime: 1684167885961 keysize: 64 valuesize: 95 sequence: -1 headerKeys:
[] key: task-7 payload:
{"state":"RUNNING","trace":null,"worker_id":"x","generation":437643}
LogAppendTime: 1684167903456 keysize: 64 valuesize: 98 sequence: -1 headerKeys:
[] key: task-7 payload:
{"state":"UNASSIGNED","trace":null,"worker_id":"x","generation":437643}
{code}
But usually, the RUNNING status should be appended after the UNASSIGNED,
because the worker coordinator will revoked the tasks before start new tasks.
Then we checked the log of our MM2 worker. And found that, during that time,
there was a task that revoked on worker-2 and started on worker-1.
Worker-1
{code:java}
[2023-05-15 09:24:45,950] INFO [Worker clientId=connect-1, groupId=__group]
Starting task task-7
(org.apache.kafka.connect.runtime.distributed.DistributedHerder)
[2023-05-15 09:24:45,951] INFO Creating task task-7
(org.apache.kafka.connect.runtime.Worker) {code}
Worker-2
{code:java}
[2023-05-15 09:24:40,922] INFO Stopping task task-7
(org.apache.kafka.connect.runtime.Worker) {code}
So I think the incorrect status was caused by the revoked task finished later
than the new started task, which made the UNASSIGNED status append to that
status topic after the RUNNING status.
After reading the code of DistributeHerder, I found that the task revoking is
running in a thread pool, the revoke operation just return after submit all the
callables. So I think even in the same worker, there is not a guarantee that
the revoke operation will always finish before the new tasks start.
{code:java}
for (final ConnectorTaskId taskId : tasks) {
callables.add(getTaskStoppingCallable(taskId));
}
// The actual timeout for graceful task/connector stop is applied in worker's
// stopAndAwaitTask/stopAndAwaitConnector methods.
startAndStop(callables);
log.info("Finished stopping tasks in preparation for rebalance"); {code}
was:
After a rebalance of our MM2 source tasks, we found there were several tasks
always in *UNASSIGNED* status, event the real tasks already started.
So we dump the payload of the status topic of Kafka Connect, and found the last
two status change is status *RUNNING* followed by status {*}UNASSIGNED{*}.
{code:java}
LogAppendTime: 1684167885961 keysize: 64 valuesize: 95 sequence: -1 headerKeys:
[] key: task-7 payload:
{"state":"RUNNING","trace":null,"worker_id":"x","generation":437643}
LogAppendTime: 1684167903456 keysize: 64 valuesize: 98 sequence: -1 headerKeys:
[] key: task-7 payload:
{"state":"UNASSIGNED","trace":null,"worker_id":"x","generation":437643}
{code}
But usually, the RUNNING status should be appended after the UNASSIGNED,
because the worker coordinator will revoked the tasks before start new tasks.
Then we checked the log of our MM2 worker. And found that, during that time,
there was a task that revoked on worker-2 and started on worker-1.
Worker-1
{code:java}
[2023-05-15 09:24:45,950] INFO [Worker clientId=connect-1, groupId=__group]
Starting task task-7
(org.apache.kafka.connect.runtime.distributed.DistributedHerder)
[2023-05-15 09:24:45,951] INFO Creating task task-7
(org.apache.kafka.connect.runtime.Worker) {code}
Worker-2
{code:java}
[2023-05-15 09:24:40,922] INFO Stopping task task-7
(org.apache.kafka.connect.runtime.Worker) {code}
So I think the incorrect status was caused by the revoked task finished later
than the new started task, which made the UNASSIGNED status append to that
status topic after the RUNNING status.
After reading the code of DistributeHerder, I found that the task revoking is
running in a thread pool, the revoke operation just return after submit all the
callables. So I think even in the same worker, there is not a guarantee that
the revoke operation will always finish before the new tasks start.
{code:java}
for (final ConnectorTaskId taskId : tasks) {
callables.add(getTaskStoppingCallable(taskId));
}
// The actual timeout for graceful task/connector stop is applied in worker's
// stopAndAwaitTask/stopAndAwaitConnector methods.
startAndStop(callables);
log.info("Finished stopping tasks in preparation for rebalance"); {code}
> Status of KafkaConnect task not correct
> ---
>
> Key: KAFKA-15005
> URL: https://issues.apache.org/jira/browse/KAFKA-15005
> Project: Kafka
> Issue Ty
[GitHub] [kafka] philipnee commented on a diff in pull request #13678: KAFKA-10337: await async commits in commitSync even if no offsets given
philipnee commented on code in PR #13678:
URL: https://github.com/apache/kafka/pull/13678#discussion_r1196816858
##
clients/src/test/java/org/apache/kafka/clients/consumer/internals/ConsumerCoordinatorTest.java:
##
@@ -668,6 +695,7 @@ public void onFailure(RuntimeException e,
RequestFuture future) {
coordinator.markCoordinatorUnknown("test cause");
consumerClient.pollNoWakeup();
assertTrue(asyncCallbackInvoked.get());
+assertEquals(coordinator.inFlightAsyncCommits.get(), 0);
Review Comment:
I was wondering if we could add this to the AfterEach, but it seems like
`testCommitAsyncWithUserAssignedType` would fail.
##
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerCoordinator.java:
##
@@ -1223,6 +1233,26 @@ public void maybeAutoCommitOffsetsAsync(long now) {
}
}
+private boolean waitForPendingAsyncCommits(Timer timer) {
Review Comment:
Hey @erikvanoosten - I took a closer look at your PR. I think your current
approach is fine.
##
clients/src/test/java/org/apache/kafka/clients/consumer/internals/ConsumerCoordinatorTest.java:
##
@@ -600,6 +601,30 @@ public void testCoordinatorNotAvailable() {
assertTrue(coordinator.coordinatorUnknown());
}
+@Test
+public void testSyncCommitWithoutOffsetsAndPendingAsyncCommit() {
Review Comment:
maybe `testEnsureCompletingAsyncCommitsWhenSyncCommitWithoutOffsets` ? I
thought the original naming sounds a bit like firing sync commit without offset
and async commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [kafka] cmccabe closed pull request #13727: MINOR: Rename handleSnapshot to handleLoadSnapshot
cmccabe closed pull request #13727: MINOR: Rename handleSnapshot to handleLoadSnapshot URL: https://github.com/apache/kafka/pull/13727 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] cmccabe commented on pull request #13727: MINOR: Rename handleSnapshot to handleLoadSnapshot
cmccabe commented on PR #13727: URL: https://github.com/apache/kafka/pull/13727#issuecomment-1551766740 merged, thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] philipnee commented on pull request #13678: KAFKA-10337: await async commits in commitSync even if no offsets given
philipnee commented on PR #13678: URL: https://github.com/apache/kafka/pull/13678#issuecomment-1551778364 @erikvanoosten - Thanks for following up on this PR, I think we are really closed here. Also apologize about the misleading comment. I left a few comments above. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] satishd commented on pull request #13535: KAFKA-9579 Fetch implementation for records in the remote storage through a specific purgatory.
satishd commented on PR #13535: URL: https://github.com/apache/kafka/pull/13535#issuecomment-1551837017 Thanks @junrao for the latest comments, addressed them with the latest commit. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (KAFKA-15006) kafka-get-offsets.sh returns 0 for earliest offset on a compacted topic
Christopher Auston created KAFKA-15006:
--
Summary: kafka-get-offsets.sh returns 0 for earliest offset on a
compacted topic
Key: KAFKA-15006
URL: https://issues.apache.org/jira/browse/KAFKA-15006
Project: Kafka
Issue Type: Bug
Affects Versions: 3.4.0
Reporter: Christopher Auston
After some compactions and when the .log is empty, it seems
like kafka-get-offsets.sh (Admin client?) should return a non-zero offset for
the earliest offset?
A simple, compacted topic:
{code:java}
kraft-combined-logs % kafka-topics.sh --bootstrap-server :9092 -topic offtest
-describeTopic: offtest TopicId: pwmz5qPGQsyo3006M_VBqw PartitionCount: 1
ReplicationFactor: 1Configs: cleanup.policy=compact,segment.bytes=16384
Topic: offtest Partition: 0Leader: 1 Replicas: 1 Isr: 1{code}
Trying to get the earliest offset after some compactions returns 0 which is not
where a consumer starts with earliest (also the 000 log segment is empty):
{code:java}
kraft-combined-logs % kafka-get-offsets.sh --bootstrap-server :9092 -topic
offtest -time -2
offtest:0:0
kraft-combined-logs % kcat -b :9092 -C -t offtest -c1 -J
{"topic":"offtest","partition":0,"offset":161,"tstype":"create","ts":1684347565956,"broker":1,"key":"key1","payload":"
value 34"}
kraft-combined-logs % ls -l offtest-0
total 41024
-rw-r--r-- 1 ca9898 staff 0 May 17 14:16 .index
-rw-r--r-- 1 ca9898 staff 0 May 17 14:16 .log
-rw-r--r-- 1 ca9898 staff 0 May 17 14:16
.timeindex
-rw-r--r-- 1 ca9898 staff 0 May 17 14:19 0081.index
-rw-r--r-- 1 ca9898 staff 202 May 17 14:19 0081.log
-rw-r--r-- 1 ca9898 staff 10 May 17 14:16 0081.snapshot
-rw-r--r-- 1 ca9898 staff 12 May 17 14:19
0081.timeindex
-rw-r--r-- 1 ca9898 staff 10485760 May 17 14:19 0162.index
-rw-r--r-- 1 ca9898 staff 6060 May 17 14:19 0162.log
-rw-r--r-- 1 ca9898 staff 10 May 17 14:19 0162.snapshot
-rw-r--r-- 1 ca9898 staff 10485756 May 17 14:19
0162.timeindex
-rw-r--r-- 1 ca9898 staff 8 May 17 13:56 leader-epoch-checkpoint
-rw-r--r-- 1 ca9898 staff 43 May 17 13:56 partition.metadata
{code}
Server console log:
{code:java}
[2023-05-17 13:50:52,461] INFO [SocketServer listenerType=BROKER, nodeId=1]
Enabling request processing. (kafka.network.SocketServer)
[2023-05-17 13:50:52,496] INFO [BrokerLifecycleManager id=1] The broker has
been unfenced. Transitioning from RECOVERY to RUNNING.
(kafka.server.BrokerLifecycleManager)
[2023-05-17 13:50:52,496] INFO [BrokerServer id=1] Transition from STARTING to
STARTED (kafka.server.BrokerServer)
[2023-05-17 13:50:52,496] INFO Kafka version: 3.4.0
(org.apache.kafka.common.utils.AppInfoParser)
[2023-05-17 13:50:52,500] INFO Kafka commitId: 2e1947d240607d53
(org.apache.kafka.common.utils.AppInfoParser)
[2023-05-17 13:50:52,500] INFO Kafka startTimeMs: 1684345852496
(org.apache.kafka.common.utils.AppInfoParser)
[2023-05-17 13:50:52,501] INFO [KafkaRaftServer nodeId=1] Kafka Server started
(kafka.server.KafkaRaftServer)
[2023-05-17 13:56:18,521] INFO [ReplicaFetcherManager on broker 1] Removed
fetcher for partitions Set(offtest-0) (kafka.server.ReplicaFetcherManager)
[2023-05-17 13:56:18,530] INFO [LogLoader partition=offtest-0,
dir=/Users/ca9898/kraft-combined-logs] Loading producer state till offset 0
with message format version 2 (kafka.log.UnifiedLog$)
[2023-05-17 13:56:18,531] INFO Created log for partition offtest-0 in
/Users/ca9898/kraft-combined-logs/offtest-0 with properties {}
(kafka.log.LogManager)
[2023-05-17 13:56:18,536] INFO [Partition offtest-0 broker=1] No checkpointed
highwatermark is found for partition offtest-0 (kafka.cluster.Partition)
[2023-05-17 13:56:18,536] INFO [Partition offtest-0 broker=1] Log loaded for
partition offtest-0 with initial high watermark 0 (kafka.cluster.Partition)
[2023-05-17 14:06:25,254] INFO [DynamicConfigPublisher nodeType=broker id=1]
Updating topic offtest with new configuration : cleanup.policy ->
compact,segment.bytes -> 16384 (kafka.server.metadata.DynamicConfigPublisher)
[2023-05-17 14:16:30,990] INFO [BrokerToControllerChannelManager broker=1
name=forwarding] Node 1 disconnected. (org.apache.kafka.clients.NetworkClient)
[2023-05-17 14:16:53,142] INFO [LocalLog partition=offtest-0,
dir=/Users/ca9898/kraft-combined-logs] Rolled new log segment at offset 81 in 2
ms. (kafka.log.LocalLog)
[2023-05-17 14:16:53,147] INFO [ProducerStateManager partition=offtest-0] Wrote
producer snapshot at offset 81 with 0 producer ids in 5 ms.
(ka
[GitHub] [kafka] JimGalasyn commented on pull request #13728: Add default configs for internal topics
JimGalasyn commented on PR #13728: URL: https://github.com/apache/kafka/pull/13728#issuecomment-1551916913 cc @mjsax  -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] pprovenzano opened a new pull request, #13729: KAFKA-14970: Dual write mode testing for SCRAM and Quota
pprovenzano opened a new pull request, #13729: URL: https://github.com/apache/kafka/pull/13729 Add two tests for dual write mode for ZK to KRaft migration. These test updating SCRAM records while in dual write mode and validate that the records are written to ZK. Fixed a bug during dual write mode where if a user is updating SCRAM records and has no quota the system will have issues. The tests explicitly validate the fix. ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (KAFKA-14970) Dual write mode testing for SCRAM and Quota
[ https://issues.apache.org/jira/browse/KAFKA-14970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Proven Provenzano updated KAFKA-14970: -- Fix Version/s: 3.5.0 > Dual write mode testing for SCRAM and Quota > --- > > Key: KAFKA-14970 > URL: https://issues.apache.org/jira/browse/KAFKA-14970 > Project: Kafka > Issue Type: Test > Components: kraft >Reporter: Proven Provenzano >Assignee: Proven Provenzano >Priority: Major > Fix For: 3.5.0 > > > SCRAM and Quota are stored together in ZK and we need better testing to > validate the dual write mode support for them. > I will add some additional tests for this. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [kafka] kirktrue commented on pull request #13591: KAFKA-14831: Illegal state errors should be fatal in transactional producer
kirktrue commented on PR #13591: URL: https://github.com/apache/kafka/pull/13591#issuecomment-1552142480 > [ea81e99](https://github.com/apache/kafka/commit/ea81e99e5980c807414651034a8c60426a158ca4) added TransitionToUninitialized. Can we confirm this has the behavior we expect? I believe this would count as a background transition. @jolshan Yes, based on the call stack... ``` Sender.runOnce() → Sender().shouldHandleAuthorizationError() → TransactionManager.transitionToUninitialized() ``` ...it's a `BACKGROUND` transition. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] artemlivshits commented on a diff in pull request #13608: KAFKA-14884: Include check transaction is still ongoing right before append
artemlivshits commented on code in PR #13608:
URL: https://github.com/apache/kafka/pull/13608#discussion_r1197145142
##
storage/src/main/java/org/apache/kafka/storage/internals/log/ProducerStateEntry.java:
##
@@ -41,18 +42,35 @@ public class ProducerStateEntry {
private int coordinatorEpoch;
private long lastTimestamp;
private OptionalLong currentTxnFirstOffset;
+
+private VerificationState verificationState;
+
+// Before any batches are associated with the entry, the tentative
sequence represents the lowest sequence seen.
+private OptionalInt tentativeSequence;
+
+public enum VerificationState {
+EMPTY,
+VERIFYING,
+VERIFIED
+}
Review Comment:
As discussed offline, this race can cause hanging transactions:
1. producer properly adds partition to transaction
2. producer has trouble sending messages, retries a few times (all messages
have the same epoch + sequence)
3. partition leader gets first message, sets state to VERIFYING, issues
request to TC
4. TC sees the open transaction, replies with success
5. producer gives up and aborts transaction
6. partition leader gets abort marker, sets state to EMPTY
7. partition leader gets second message, sets state to VERIFYING, issues
request to TC
8. successful reply from step 4 finally arrives, sees the state is
VERIFYING, sets to VERIFIED
9. the first message is written on top of abort marker, resulting in
"hanging" transaction
the issue is that transition VERIFYING - EMPTY - VERIFYING is the same as
VERIFYING, the classic ABA problem https://en.wikipedia.org/wiki/ABA_problem
##
storage/src/main/java/org/apache/kafka/storage/internals/log/ProducerStateEntry.java:
##
@@ -104,27 +122,63 @@ public boolean maybeUpdateProducerEpoch(short
producerEpoch) {
}
}
+public boolean maybeUpdateProducerHigherEpoch(short producerEpoch) {
+if (this.producerEpoch < producerEpoch) {
+batchMetadata.clear();
+this.producerEpoch = producerEpoch;
+return true;
+} else {
+return false;
+}
+}
+
+// We only set tentative sequence if no batches have been written to the
log. It is used to avoid OutOfOrderSequenceExceptions
+// when we saw a lower sequence during transaction verification. We will
update the sequence when there is no batch metadata if:
+// a) There is no tentative sequence yet
+// b) A lower sequence for the same epoch is seen and should thereby
block records after that
+// c) A higher producer epoch is found that will reset the lowest seen
sequence
+public void maybeUpdateTentativeSequence(int sequence, short
producerEpoch) {
+if (batchMetadata.isEmpty() &&
Review Comment:
We only need tentative state when we don't have an entry for something that
we've written. I'm not sure if doing something that we don't need so that we
don't rely on it later makes the code simpler :-).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [kafka] AmareshB commented on a diff in pull request #6247: KAFKA-7466: Add IncrementalAlterConfigs API (KIP-339)
AmareshB commented on code in PR #6247:
URL: https://github.com/apache/kafka/pull/6247#discussion_r1197203647
##
clients/src/main/java/org/apache/kafka/clients/admin/AdminClient.java:
##
@@ -388,6 +388,48 @@ public AlterConfigsResult alterConfigs(Map configs) {
*/
public abstract AlterConfigsResult alterConfigs(Map configs, AlterConfigsOptions options);
+/**
+ * Incrementally updates the configuration for the specified resources
with default options.
+ *
+ * This is a convenience method for #{@link
AdminClient#incrementalAlterConfigs(Map, IncrementalAlterConfigsOptions)} with
default options.
+ * See the overload for more details.*
+ *
+ * This operation is supported by brokers with version 2.3.0 or higher.
+ *
+ * @param configs The resources with their configs
+ * @returnThe IncrementalAlterConfigsResult
+ */
+public IncrementalAlterConfigsResult
incrementalAlterConfigs(Map> configs)
{
+return incrementalAlterConfigs(configs, new
IncrementalAlterConfigsOptions());
+}
+
+
+/**
+ * Incrementally update the configuration for the specified resources.
+ *
+ * Updates are not transactional so they may succeed for some resources
while fail for others. The configs for
Review Comment:
KIP mentions this is `transactional functionality` but the code here says
the opposite. Can you please clarify ?
From the KIP in proposed changes section - Similar to existing alterConfigs
API, we will to keep the "transactionality" of updating several configs for
the same ConfigResource at once. We guarantee that we never do a partial update
of a collection of configs
for a ConfigResource from a single request.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [kafka] AmareshB commented on a diff in pull request #6247: KAFKA-7466: Add IncrementalAlterConfigs API (KIP-339)
AmareshB commented on code in PR #6247:
URL: https://github.com/apache/kafka/pull/6247#discussion_r1197203647
##
clients/src/main/java/org/apache/kafka/clients/admin/AdminClient.java:
##
@@ -388,6 +388,48 @@ public AlterConfigsResult alterConfigs(Map configs) {
*/
public abstract AlterConfigsResult alterConfigs(Map configs, AlterConfigsOptions options);
+/**
+ * Incrementally updates the configuration for the specified resources
with default options.
+ *
+ * This is a convenience method for #{@link
AdminClient#incrementalAlterConfigs(Map, IncrementalAlterConfigsOptions)} with
default options.
+ * See the overload for more details.*
+ *
+ * This operation is supported by brokers with version 2.3.0 or higher.
+ *
+ * @param configs The resources with their configs
+ * @returnThe IncrementalAlterConfigsResult
+ */
+public IncrementalAlterConfigsResult
incrementalAlterConfigs(Map> configs)
{
+return incrementalAlterConfigs(configs, new
IncrementalAlterConfigsOptions());
+}
+
+
+/**
+ * Incrementally update the configuration for the specified resources.
+ *
+ * Updates are not transactional so they may succeed for some resources
while fail for others. The configs for
Review Comment:
KIP mentions this is `transactional functionality` but the code here says
the opposite. Can you please clarify ?
From the KIP in proposed changes section -
`Similar to existing alterConfigs API, we will to keep the
"transactionality" of updating several configs for the same ConfigResource at
once. We guarantee that we never do a partial update of a collection of configs
for a ConfigResource from a single request. `
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [kafka] satishd merged pull request #13535: KAFKA-9579 Fetch implementation for records in the remote storage through a specific purgatory.
satishd merged PR #13535: URL: https://github.com/apache/kafka/pull/13535 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (KAFKA-14939) Only expose ZkMigrationState metric if metadata.version supports it
[ https://issues.apache.org/jira/browse/KAFKA-14939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] HaiyuanZhao reassigned KAFKA-14939: --- Assignee: HaiyuanZhao > Only expose ZkMigrationState metric if metadata.version supports it > --- > > Key: KAFKA-14939 > URL: https://issues.apache.org/jira/browse/KAFKA-14939 > Project: Kafka > Issue Type: Sub-task >Affects Versions: 3.5.0 >Reporter: David Arthur >Assignee: HaiyuanZhao >Priority: Minor > > We should only expose the KafkaController.ZkMigrationState JMX metric if the > cluster is running on a metadata.version that supports migrations. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-14846) Fix overly large record batches in ZkMigrationClient
[ https://issues.apache.org/jira/browse/KAFKA-14846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] HaiyuanZhao reassigned KAFKA-14846: --- Assignee: HaiyuanZhao > Fix overly large record batches in ZkMigrationClient > > > Key: KAFKA-14846 > URL: https://issues.apache.org/jira/browse/KAFKA-14846 > Project: Kafka > Issue Type: Sub-task >Affects Versions: 3.4.0 >Reporter: Colin McCabe >Assignee: HaiyuanZhao >Priority: Major > > ZkMigrationClient should not create overly large record batches -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-14833) ZK migration cannot start if a ZK controller was not previously elected
[
https://issues.apache.org/jira/browse/KAFKA-14833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
HaiyuanZhao reassigned KAFKA-14833:
---
Assignee: HaiyuanZhao
> ZK migration cannot start if a ZK controller was not previously elected
> ---
>
> Key: KAFKA-14833
> URL: https://issues.apache.org/jira/browse/KAFKA-14833
> Project: Kafka
> Issue Type: Sub-task
> Components: kraft
>Affects Versions: 3.4.0
>Reporter: David Arthur
>Assignee: HaiyuanZhao
>Priority: Minor
>
> During the BecomeZkController event in KRaftMigrationDriver, an error occurs
> if there was not data in the /controller_epoch znode.
> {code}
> [2023-03-22 14:33:06,082] ERROR Had an exception in BecomeZkControllerEvent
> (org.apache.kafka.metadata.migration.KRaftMigrationDriver)
> java.lang.IllegalStateException: Cannot register KRaft controller 3001 as the
> active controller since there is no ZK controller epoch present.
> at
> kafka.zk.KafkaZkClient.tryRegisterKRaftControllerAsActiveController(KafkaZkClient.scala:201)
> at
> kafka.zk.ZkMigrationClient.claimControllerLeadership(ZkMigrationClient.scala:60)
> at
> org.apache.kafka.metadata.migration.KRaftMigrationDriver.apply(KRaftMigrationDriver.java:143)
> at
> org.apache.kafka.metadata.migration.KRaftMigrationDriver.access$800(KRaftMigrationDriver.java:57)
> at
> org.apache.kafka.metadata.migration.KRaftMigrationDriver$BecomeZkControllerEvent.run(KRaftMigrationDriver.java:357)
> at
> org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:121)
> at
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:201)
> at
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:175)
> at java.lang.Thread.run(Thread.java:750)
> {code}
> This mainly affects tests when we start a KRaft quorum first, and then the
> (new) ZK cluster. The migration driver will kick off when the ZK brokers
> register but it will race with KafkaController on the /controller_epoch
> creation.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [kafka] aaron-ai commented on pull request #13710: MINOR: Fix the outdated comments of ConfigDef
aaron-ai commented on PR #13710: URL: https://github.com/apache/kafka/pull/13710#issuecomment-1552382217 @jlprat @showuon @C0urante @dajac Could you please take a look? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] vamossagar12 commented on pull request #13726: KAFKA-13109: SourceTask#poll not enforcing retries in case of RetriableException
vamossagar12 commented on PR #13726: URL: https://github.com/apache/kafka/pull/13726#issuecomment-1552416862 Test failures seem unrelated. Couple of failing connect related tests are being fixed via separate JIRA tickets -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] DengShuaiSimon opened a new pull request, #13730: Feature/shuai add comment
DengShuaiSimon opened a new pull request, #13730: URL: https://github.com/apache/kafka/pull/13730 *More detailed description of your change, if necessary. The PR title and PR message become the squashed commit message, so use a separate comment to ping reviewers.* *Summary of testing strategy (including rationale) for the feature or bug fix. Unit and/or integration tests are expected for any behaviour change and system tests should be considered for larger changes.* ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
