[jira] [Created] (KAFKA-16181) Use incrementalAlterConfigs when updating broker configs by kafka-configs.sh
Deng Ziming created KAFKA-16181: --- Summary: Use incrementalAlterConfigs when updating broker configs by kafka-configs.sh Key: KAFKA-16181 URL: https://issues.apache.org/jira/browse/KAFKA-16181 Project: Kafka Issue Type: Improvement Reporter: Deng Ziming Assignee: Deng Ziming -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-15619) Deleted topics will come back again
[
https://issues.apache.org/jira/browse/KAFKA-15619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Deng Ziming resolved KAFKA-15619.
-
Assignee: Deng Ziming
Resolution: Invalid
This has been fixed after setting auto.create.topics.enable=false.
> Deleted topics will come back again
> ---
>
> Key: KAFKA-15619
> URL: https://issues.apache.org/jira/browse/KAFKA-15619
> Project: Kafka
> Issue Type: Improvement
>Affects Versions: 3.5.0, 3.5.1
>Reporter: Deng Ziming
>Assignee: Deng Ziming
>Priority: Major
>
> Deleted topics will come back again in Apache Spark structured streaming
> stress test after upgrade Kafka from 3.4.0 to 3.5.0, related ticket is:
> https://issues.apache.org/jira/browse/SPARK-45529 , the test randomly
> starts/stops/adds data/add partitions/delete topic/add topic/checks the
> result in a loop, I finally found that a deleted topic will come back again
> after some time.
> By constantly reseting the head of branch-3.5 and using {{gradlew install}}
> to repackage and rerunning of the stress test, I have basically inferred that
> this bug comes from [https://github.com/apache/kafka/pull/12590]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KAFKA-13005) Support JBOD in kraft mode
[ https://issues.apache.org/jira/browse/KAFKA-13005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deng Ziming resolved KAFKA-13005. - Resolution: Duplicate > Support JBOD in kraft mode > -- > > Key: KAFKA-13005 > URL: https://issues.apache.org/jira/browse/KAFKA-13005 > Project: Kafka > Issue Type: Improvement >Reporter: Colin McCabe >Assignee: Deng Ziming >Priority: Major > Labels: kip-500 > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-15390) FetchResponse.preferredReplica may contains fenced replica in KRaft mode
[ https://issues.apache.org/jira/browse/KAFKA-15390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deng Ziming resolved KAFKA-15390. - Fix Version/s: 3.6.0 Resolution: Fixed > FetchResponse.preferredReplica may contains fenced replica in KRaft mode > > > Key: KAFKA-15390 > URL: https://issues.apache.org/jira/browse/KAFKA-15390 > Project: Kafka > Issue Type: Bug >Reporter: Deng Ziming >Assignee: Deng Ziming >Priority: Major > Fix For: 3.6.0 > > > `KRaftMetadataCache.getPartitionReplicaEndpoints` will return a fenced broker > id. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-15619) Kafka
Deng Ziming created KAFKA-15619: --- Summary: Kafka Key: KAFKA-15619 URL: https://issues.apache.org/jira/browse/KAFKA-15619 Project: Kafka Issue Type: Improvement Affects Versions: 3.5.1, 3.5.0 Reporter: Deng Ziming Deleted topics will come back again in Spark structured streaming test after upgrade Kafka from 3.4.0 to 3.5.0, related ticket is: https://issues.apache.org/jira/browse/SPARK-45529 I have basically inferred that this bug comes from https://github.com/apache/kafka/pull/12590 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-15566) Klaky tests in FetchRequestTest.scala in KRaft mode
Deng Ziming created KAFKA-15566: --- Summary: Klaky tests in FetchRequestTest.scala in KRaft mode Key: KAFKA-15566 URL: https://issues.apache.org/jira/browse/KAFKA-15566 Project: Kafka Issue Type: Improvement Reporter: Deng Ziming |[https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-14295/4/#showFailuresLink] [Build / JDK 11 and Scala 2.13 / kafka.server.FetchRequestTest.testLastFetchedEpochValidation(String).quorum=kraft|https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-14295/4/testReport/junit/kafka.server/FetchRequestTest/Build___JDK_11_and_Scala_2_13___testLastFetchedEpochValidation_String__quorum_kraft/] [Build / JDK 11 and Scala 2.13 / kafka.server.FetchRequestTest.testLastFetchedEpochValidationV12(String).quorum=kraft|https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-14295/4/testReport/junit/kafka.server/FetchRequestTest/Build___JDK_11_and_Scala_2_13___testLastFetchedEpochValidationV12_String__quorum_kraft/] [Build / JDK 11 and Scala 2.13 / kafka.server.FetchRequestTest.testFetchWithPartitionsWithIdError(String).quorum=kraft|https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-14295/4/testReport/junit/kafka.server/FetchRequestTest/Build___JDK_11_and_Scala_2_13___testFetchWithPartitionsWithIdError_String__quorum_kraft_2/] [Build / JDK 11 and Scala 2.13 / kafka.server.FetchRequestTest.testLastFetchedEpochValidation(String).quorum=kraft|https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-14295/4/testReport/junit/kafka.server/FetchRequestTest/Build___JDK_11_and_Scala_2_13___testLastFetchedEpochValidation_String__quorum_kraft_2/] [Build / JDK 11 and Scala 2.13 / kafka.server.FetchRequestTest.testLastFetchedEpochValidationV12(String).quorum=kraft|https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-14295/4/testReport/junit/kafka.server/FetchRequestTest/Build___JDK_11_and_Scala_2_13___testLastFetchedEpochValidationV12_String__quorum_kraft_2/]| -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-15315) Use getOrDefault rather than get
[ https://issues.apache.org/jira/browse/KAFKA-15315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deng Ziming resolved KAFKA-15315. - Resolution: Fixed > Use getOrDefault rather than get > > > Key: KAFKA-15315 > URL: https://issues.apache.org/jira/browse/KAFKA-15315 > Project: Kafka > Issue Type: Improvement > Components: clients >Reporter: roon >Priority: Trivial > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Reopened] (KAFKA-15140) Improve TopicCommandIntegrationTest to be less flaky
[
https://issues.apache.org/jira/browse/KAFKA-15140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Deng Ziming reopened KAFKA-15140:
-
This is still flaky, it seems the internal topic is also under min isr
partitions.
https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-13562/23/testReport/junit/kafka.admin/TopicCommandIntegrationTest/Build___JDK_8_and_Scala_2_12___testDescribeUnderMinIsrPartitionsMixed_String__quorum_kraft/
> Improve TopicCommandIntegrationTest to be less flaky
>
>
> Key: KAFKA-15140
> URL: https://issues.apache.org/jira/browse/KAFKA-15140
> Project: Kafka
> Issue Type: Test
> Components: unit tests
>Reporter: Divij Vaidya
>Assignee: Lan Ding
>Priority: Minor
> Labels: newbie
> Fix For: 3.6.0, 3.5.1
>
>
> *This is a good Jira for folks who are new to contributing to Kafka.*
> Tests in TopicCommandIntegrationTest get flaky from time to time. The
> objective of the task is to make them more robust by doing the following:
> 1. Replace the usage {-}createAndWaitTopic{-}() adminClient.createTopics()
> method and other places where were are creating a topic (without waiting)
> with
> TestUtils.createTopicWithAdmin(). The latter method already contains the
> functionality to create a topic and wait for metadata to sync up.
> 2. Replace the number 6 at places such as
> "adminClient.createTopics(
> Collections.singletonList(new NewTopic("foo_bar", 1, 6.toShort)))" with a
> meaningful constant.
> 3. Add logs if an assertion fails, for example, lines such as "
> assertTrue(rows(0).startsWith(s"\tTopic: $testTopicName"), output)" should
> have a third argument which prints the actual output printed so that we can
> observe in the test logs on what was the output when assertion failed.
> 4. Replace occurrences of "\n" with System.lineSeparator() which is platform
> independent
> 5. We should wait for reassignment to complete whenever we are re-assigning
> partitions using alterconfig before we call describe to validate it. We could
> use
> TestUtils.waitForAllReassignmentsToComplete()
> *Motivation of this task*
> Try to fix the flaky test behaviour such as observed in
> [https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-13924/5/testReport/junit/kafka.admin/TopicCommandIntegrationTest/Build___JDK_11_and_Scala_2_13___testDescribeUnderMinIsrPartitionsMixed_String__quorum_zk/]
>
> {noformat}
> org.opentest4j.AssertionFailedError: expected: but was:
> at
> app//org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
> at
> app//org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
> at app//org.junit.jupiter.api.AssertTrue.failNotTrue(AssertTrue.java:63)
> at app//org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:36)
> at app//org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:31)
> at app//org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:180)
> at
> app//kafka.admin.TopicCommandIntegrationTest.testDescribeUnderMinIsrPartitionsMixed(TopicCommandIntegrationTest.scala:794){noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KAFKA-15286) Migrate ApiVersion related code to kraft
[ https://issues.apache.org/jira/browse/KAFKA-15286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deng Ziming resolved KAFKA-15286. - Resolution: Fixed > Migrate ApiVersion related code to kraft > > > Key: KAFKA-15286 > URL: https://issues.apache.org/jira/browse/KAFKA-15286 > Project: Kafka > Issue Type: Task >Reporter: Deng Ziming >Assignee: Deng Ziming >Priority: Major > > In many places involving ApiVersion, we only support zk, we should move it > forward to kraft. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-15390) FetchResponse.preferredReplica may contains fenced replica in KRaft mode
Deng Ziming created KAFKA-15390: --- Summary: FetchResponse.preferredReplica may contains fenced replica in KRaft mode Key: KAFKA-15390 URL: https://issues.apache.org/jira/browse/KAFKA-15390 Project: Kafka Issue Type: Bug Reporter: Deng Ziming Assignee: Deng Ziming `KRaftMetadataCache.getPartitionReplicaEndpoints` will return a fenced broker id. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-15371) MetadataShell is stuck when bootstrapping
Deng Ziming created KAFKA-15371:
---
Summary: MetadataShell is stuck when bootstrapping
Key: KAFKA-15371
URL: https://issues.apache.org/jira/browse/KAFKA-15371
Project: Kafka
Issue Type: Bug
Affects Versions: 3.5.1
Reporter: Deng Ziming
Attachments: image-2023-08-17-10-35-01-039.png,
image-2023-08-17-10-35-36-067.png
I downloaded the 3.5.1 package and startup it, then use metadata shell to
inspect the data
{code:java}
// shell
bin/kafka-metadata-shell.sh --snapshot
/tmp/kraft-combined-logs/__cluster_metadata-0/.log {code}
Then process will stuck at loading.
!image-2023-08-17-10-35-36-067.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KAFKA-15354) Partition leader is not evenly distributed in kraft mode
Deng Ziming created KAFKA-15354:
---
Summary: Partition leader is not evenly distributed in kraft mode
Key: KAFKA-15354
URL: https://issues.apache.org/jira/browse/KAFKA-15354
Project: Kafka
Issue Type: Improvement
Reporter: Deng Ziming
In StripedReplicaPlacerTest, we can create a test below to reproduce this bug.
{code:java}
// code placeholder
@Test
public void testReplicaDistribution() {
MockRandom random = new MockRandom();
StripedReplicaPlacer placer = new StripedReplicaPlacer(random);
TopicAssignment assignment = place(placer, 0, 4, (short) 2, Arrays.asList(
new UsableBroker(0, Optional.of("0"), false),
new UsableBroker(1, Optional.of("0"), false),
new UsableBroker(2, Optional.of("1"), false),
new UsableBroker(3, Optional.of("1"), false)));
System.out.println(assignment);
} {code}
In StripedReplicaPlacer, we only ensure leader are distributed evenly across
racks, but we didn't ensure leader are evenly distributed across nodes. in the
test above, we have 4 node: 1 2 3 4, and create 4 partitions but the leaders
are 1 2 1 2. while in zk mode, this is ensured, see
https://cwiki.apache.org/confluence/display/KAFKA/KIP-36+Rack+aware+replica+assignment
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KAFKA-13836) Improve KRaft broker heartbeat logic
[ https://issues.apache.org/jira/browse/KAFKA-13836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deng Ziming resolved KAFKA-13836. - Resolution: Won't Fix > Improve KRaft broker heartbeat logic > > > Key: KAFKA-13836 > URL: https://issues.apache.org/jira/browse/KAFKA-13836 > Project: Kafka > Issue Type: Improvement >Reporter: Deng Ziming >Assignee: Deng Ziming >Priority: Major > > # Don't advertise an offset to the controller until it has been published > # only unfence a broker when it has seen it's own registration -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-15289) Support KRaft mode in RequestQuotaTest
[ https://issues.apache.org/jira/browse/KAFKA-15289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deng Ziming resolved KAFKA-15289. - Resolution: Fixed > Support KRaft mode in RequestQuotaTest > -- > > Key: KAFKA-15289 > URL: https://issues.apache.org/jira/browse/KAFKA-15289 > Project: Kafka > Issue Type: Sub-task >Reporter: Deng Ziming >Priority: Major > Labels: newbee > > we are calling `zkBrokerApis` in RequestQuotaTest, we should ensure kraft > broker apis are also supported, so use clientApis as far as possible.use > zkBrokerApis.clientApis instead of ApiKeys.zkBrokerApis. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-15340) Test request quota for kraft controller apis
Deng Ziming created KAFKA-15340: --- Summary: Test request quota for kraft controller apis Key: KAFKA-15340 URL: https://issues.apache.org/jira/browse/KAFKA-15340 Project: Kafka Issue Type: Improvement Components: kraft, unit tests Reporter: Deng Ziming The RequestQuotaTest only tests request quota for kraft broker apis and zk broker apis, we should also test kraft controller apis. further more, maybe there are others tests we need to complement for kraft controller apis(not only broker apis). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-15334) DescribeQuorum should not be seen as cluster action
Deng Ziming created KAFKA-15334: --- Summary: DescribeQuorum should not be seen as cluster action Key: KAFKA-15334 URL: https://issues.apache.org/jira/browse/KAFKA-15334 Project: Kafka Issue Type: Bug Reporter: Deng Ziming -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-15287) Change NodeApiVersions.create() to contains both apis of zk and kraft broker
[ https://issues.apache.org/jira/browse/KAFKA-15287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deng Ziming resolved KAFKA-15287. - Resolution: Fixed > Change NodeApiVersions.create() to contains both apis of zk and kraft broker > - > > Key: KAFKA-15287 > URL: https://issues.apache.org/jira/browse/KAFKA-15287 > Project: Kafka > Issue Type: Sub-task >Reporter: Deng Ziming >Priority: Major > Labels: newbee > > We are using ApiKeys.zkBrokerApis() when calling NodeApiVersions.create(), > this means we only support zk broker apis. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-15288) Change BrokerApiVersionsCommandTest to support kraft mode
[ https://issues.apache.org/jira/browse/KAFKA-15288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deng Ziming resolved KAFKA-15288. - Resolution: Fixed > Change BrokerApiVersionsCommandTest to support kraft mode > - > > Key: KAFKA-15288 > URL: https://issues.apache.org/jira/browse/KAFKA-15288 > Project: Kafka > Issue Type: Sub-task >Reporter: Deng Ziming >Priority: Minor > > Currently we only test zk mode for BrokerApiVersionsCommand -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-15289) Use zkBrokerApis.clientApis instead of ApiKeys.zkBrokerApis in most cases
Deng Ziming created KAFKA-15289: --- Summary: Use zkBrokerApis.clientApis instead of ApiKeys.zkBrokerApis in most cases Key: KAFKA-15289 URL: https://issues.apache.org/jira/browse/KAFKA-15289 Project: Kafka Issue Type: Sub-task Reporter: Deng Ziming In most Test cases, we are calling `zkBrokerApis`, we should ensure kraft broker apis are also supported, so use clientApis as far as possible. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-15288) Change BrokerApiVersionsCommandTest to support kraft mode
Deng Ziming created KAFKA-15288: --- Summary: Change BrokerApiVersionsCommandTest to support kraft mode Key: KAFKA-15288 URL: https://issues.apache.org/jira/browse/KAFKA-15288 Project: Kafka Issue Type: Sub-task Reporter: Deng Ziming Currently -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-15287) Change NodeApiVersions.create() to contains both apis of zk and kraft broker
Deng Ziming created KAFKA-15287: --- Summary: Change NodeApiVersions.create() to contains both apis of zk and kraft broker Key: KAFKA-15287 URL: https://issues.apache.org/jira/browse/KAFKA-15287 Project: Kafka Issue Type: Sub-task Reporter: Deng Ziming We are using ApiKeys.zkBrokerApis() when calling NodeApiVersions.create(), this means we only support zk broker apis. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-15286) Migrate ApiVersion related code to kraft
Deng Ziming created KAFKA-15286: --- Summary: Migrate ApiVersion related code to kraft Key: KAFKA-15286 URL: https://issues.apache.org/jira/browse/KAFKA-15286 Project: Kafka Issue Type: Task Reporter: Deng Ziming Assignee: Deng Ziming In many places involving ApiVersion, we only support zk, we should move it forward to kraft. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-15036) Kraft leader change fails when invoking getFinalizedFeatures
[
https://issues.apache.org/jira/browse/KAFKA-15036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Deng Ziming resolved KAFKA-15036.
-
Resolution: Fixed
> Kraft leader change fails when invoking getFinalizedFeatures
>
>
> Key: KAFKA-15036
> URL: https://issues.apache.org/jira/browse/KAFKA-15036
> Project: Kafka
> Issue Type: Improvement
>Reporter: Deng Ziming
>Assignee: Deng Ziming
>Priority: Major
>
> When kraft leader changes, we can receiving a error as follows:
>
> {{[2023-05-24 18:00:02,898] WARN [QuorumController id=3002]
> getFinalizedFeatures: failed with unknown server exception RuntimeException
> in 271 us. The controller is already in standby mode.
> (org.apache.kafka.controller.QuorumController)
> java.lang.RuntimeException: No in-memory snapshot for epoch 9. Snapshot
> epochs are:
> at
> org.apache.kafka.timeline.SnapshotRegistry.getSnapshot(SnapshotRegistry.java:173)
> at
> org.apache.kafka.timeline.SnapshotRegistry.iterator(SnapshotRegistry.java:131)
> at org.apache.kafka.timeline.TimelineObject.get(TimelineObject.java:69)
> at
> org.apache.kafka.controller.FeatureControlManager.finalizedFeatures(FeatureControlManager.java:303)
> at
> org.apache.kafka.controller.QuorumController.lambda$finalizedFeatures$16(QuorumController.java:2016)
> at
> org.apache.kafka.controller.QuorumController$ControllerReadEvent.run(QuorumController.java:546)
> at
> org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:127)
> at
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
> at
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
> at java.base/java.lang.Thread.run(Thread.java:829)}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KAFKA-15065) ApiVersionRequest is not properly handled in Sasl ControllerServer
Deng Ziming created KAFKA-15065:
---
Summary: ApiVersionRequest is not properly handled in Sasl
ControllerServer
Key: KAFKA-15065
URL: https://issues.apache.org/jira/browse/KAFKA-15065
Project: Kafka
Issue Type: Improvement
Reporter: Deng Ziming
In KAFKA-14291 we add finalizedFeatures in ApiVersionResponse, also change the
`apiVersionResponse` method to throw exception:
{code:java}
override def apiVersionResponse(requestThrottleMs: Int): ApiVersionsResponse = {
throw new UnsupportedOperationException("This method is not supported in
SimpleApiVersionManager, use apiVersionResponse(throttleTimeMs,
finalizedFeatures, epoch) instead")
} {code}
but this method is used in SocketServer:
{code:java}
private[network] val selector = createSelector(
ChannelBuilders.serverChannelBuilder(
listenerName,
listenerName == config.interBrokerListenerName,
securityProtocol,
config,
credentialProvider.credentialCache,
credentialProvider.tokenCache,
time,
logContext,
() => apiVersionManager.apiVersionResponse(throttleTimeMs = 0)
)
) {code}
And this method will be invoked in `SaslServerAuthenticator.authenticate` and
will stop the process.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KAFKA-15036) Kraft leader change fails when invoking getFinalizedFeatures
Deng Ziming created KAFKA-15036:
---
Summary: Kraft leader change fails when invoking
getFinalizedFeatures
Key: KAFKA-15036
URL: https://issues.apache.org/jira/browse/KAFKA-15036
Project: Kafka
Issue Type: Improvement
Reporter: Deng Ziming
When kraft leader changes, we can receiving a error as follows:
{{[2023-05-24 18:00:02,898] WARN [QuorumController id=3002]
getFinalizedFeatures: failed with unknown server exception RuntimeException in
271 us. The controller is already in standby mode.
(org.apache.kafka.controller.QuorumController)
java.lang.RuntimeException: No in-memory snapshot for epoch 9. Snapshot epochs
are:
at
org.apache.kafka.timeline.SnapshotRegistry.getSnapshot(SnapshotRegistry.java:173)
at
org.apache.kafka.timeline.SnapshotRegistry.iterator(SnapshotRegistry.java:131)
at org.apache.kafka.timeline.TimelineObject.get(TimelineObject.java:69)
at
org.apache.kafka.controller.FeatureControlManager.finalizedFeatures(FeatureControlManager.java:303)
at
org.apache.kafka.controller.QuorumController.lambda$finalizedFeatures$16(QuorumController.java:2016)
at
org.apache.kafka.controller.QuorumController$ControllerReadEvent.run(QuorumController.java:546)
at
org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:127)
at
org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
at
org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
at java.base/java.lang.Thread.run(Thread.java:829)}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KAFKA-14989) Flaky test TransactionsTest.testFailureToFenceEpoch
Deng Ziming created KAFKA-14989: --- Summary: Flaky test TransactionsTest.testFailureToFenceEpoch Key: KAFKA-14989 URL: https://issues.apache.org/jira/browse/KAFKA-14989 Project: Kafka Issue Type: Improvement Reporter: Deng Ziming -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Reopened] (KAFKA-14291) KRaft: ApiVersionsResponse doesn't have finalizedFeatures and finalizedFeatureEpoch in KRaft mode
[
https://issues.apache.org/jira/browse/KAFKA-14291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Deng Ziming reopened KAFKA-14291:
-
Assignee: Deng Ziming
> KRaft: ApiVersionsResponse doesn't have finalizedFeatures and
> finalizedFeatureEpoch in KRaft mode
> -
>
> Key: KAFKA-14291
> URL: https://issues.apache.org/jira/browse/KAFKA-14291
> Project: Kafka
> Issue Type: Bug
> Components: kraft
>Reporter: Akhilesh Chaganti
>Assignee: Deng Ziming
>Priority: Critical
>
> https://github.com/apache/kafka/blob/d834947ae7abc8a9421d741e742200bb36f51fb3/core/src/main/scala/kafka/server/ApiVersionManager.scala#L53
> ```
> class SimpleApiVersionManager(
> val listenerType: ListenerType,
> val enabledApis: collection.Set[ApiKeys],
> brokerFeatures: Features[SupportedVersionRange]
> ) extends ApiVersionManager {
> def this(listenerType: ListenerType) = {
> this(listenerType, ApiKeys.apisForListener(listenerType).asScala,
> BrokerFeatures.defaultSupportedFeatures())
> }
> private val apiVersions =
> ApiVersionsResponse.collectApis(enabledApis.asJava)
> override def apiVersionResponse(requestThrottleMs: Int):
> ApiVersionsResponse = {
> ApiVersionsResponse.createApiVersionsResponse(requestThrottleMs,
> apiVersions, brokerFeatures)
> }
> }
> ```
> ApiVersionManager for KRaft doesn't add the finalizedFeatures and
> finalizedFeatureEpoch to the ApiVersionsResponse.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KAFKA-14689) Kafka won't start on Windows in KRaft mode
[
https://issues.apache.org/jira/browse/KAFKA-14689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Deng Ziming resolved KAFKA-14689.
-
Resolution: Duplicate
This is the same with KAFKA-14273
> Kafka won't start on Windows in KRaft mode
> --
>
> Key: KAFKA-14689
> URL: https://issues.apache.org/jira/browse/KAFKA-14689
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 3.4.0
> Environment: Microsoft Windows 10 Pro
> 10.0.19045 Build 19045
> OpenJDK build 17.0.2
>Reporter: Kedar Joshi
>Priority: Major
> Attachments: kafka-log.txt, server.properties
>
>
> Kafka 3.4 is unable to start in KRaft mode on Windows 10.
> Kafka is started in Kafka directory with -
> {code:java}
> bin\windows\kafka-server-start.bat .\config\kraft\server.properties {code}
> but it always fails with following exception -
> {code:java}
> Caused by: java.nio.file.FileSystemException:
> D:\LocationGuru\Servers\Kafka-3.4\.\tmp\kraft-combined-logs\__cluster_metadata-0\quorum-state.tmp
> ->
> D:\LocationGuru\Servers\Kafka-3.4\.\tmp\kraft-combined-logs\__cluster_metadata-0\quorum-state:
> The process cannot access the file because it is being used by another
> process
> at
> java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:92)
> at
> java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103)
> at java.base/sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:403)
> at
> java.base/sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:293)
> at java.base/java.nio.file.Files.move(Files.java:1430)
> at
> org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:949)
> at
> org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:932)
> at
> org.apache.kafka.raft.FileBasedStateStore.writeElectionStateToFile(FileBasedStateStore.java:152)
> ... 15 more {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KAFKA-14380) consumer should refresh preferred read replica on metadata update
[ https://issues.apache.org/jira/browse/KAFKA-14380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deng Ziming resolved KAFKA-14380. - Resolution: Duplicate duplicated with KAFKA-14379 > consumer should refresh preferred read replica on metadata update > - > > Key: KAFKA-14380 > URL: https://issues.apache.org/jira/browse/KAFKA-14380 > Project: Kafka > Issue Type: Bug >Reporter: Jeff Kim >Assignee: Jeff Kim >Priority: Major > > The consumer (fetcher) clears the preferred read replica only on two > conditions: > # the consumer receives an OFFSET_OUT_OF_RANGE error > # the follower does not exist in the client's metadata (i.e., offline) > # preferred read replica value expires (5 minutes) > For other errors, it will continue to reach to the possibly unavailable > follower and only after 5 minutes will it clear preferred read replica and go > back to the leader. > A specific example is when a partition is reassigned. the consumer will get > NOT_LEADER_OR_FOLLOWER which triggers a metadata update but the preferred > read replica will not be refreshed as the follower is still online. it will > continue to reach out to the old follower until the preferred read replica > expires. > the consumer can instead clear its preferred read replica whenever it updates > its metadata. so when the consumer receives NOT_LEADER_OR_FOLLOWER in the > scenario above it can find the new preferred read replica by fetching from > the leader without waiting for the old value to expire. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-14378) consumer should refresh preferred read replica on update metadata
[ https://issues.apache.org/jira/browse/KAFKA-14378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deng Ziming resolved KAFKA-14378. - Resolution: Duplicate duplicated with KAFKA-14379 > consumer should refresh preferred read replica on update metadata > - > > Key: KAFKA-14378 > URL: https://issues.apache.org/jira/browse/KAFKA-14378 > Project: Kafka > Issue Type: Bug >Reporter: Jeff Kim >Assignee: Jeff Kim >Priority: Major > > The consumer (fetcher) refreshes the preferred read replica only on three > conditions: > # the consumer receives an OFFSET_OUT_OF_RANGE error > # the follower does not exist in the client's metadata (i.e., offline) > # after metadata.max.age.ms (5 min default) > For other errors, it will continue to reach to the possibly unavailable > follower and only after 5 minutes will it refresh the preferred read replica > and go back to the leader. > A specific example is when a partition is reassigned. the consumer will get > NOT_LEADER_OR_FOLLOWER which triggers a metadata update but the preferred > read replica will not be refreshed as the follower is still online. it will > continue to reach out to the old follower until the preferred read replica > expires. > the consumer can instead refresh its preferred read replica whenever it makes > a metadata update request. so when the consumer receives i.e. > NOT_LEADER_OR_FOLLOWER it can find the new preferred read replica without > waiting for the expiration. -- This message was sent by Atlassian Jira (v8.20.10#820010)
