[jira] [Commented] (KAFKA-17056) Convert producer state metadata schemas to use generated protocol
[ https://issues.apache.org/jira/browse/KAFKA-17056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17861003#comment-17861003 ] Kuan Po Tseng commented on KAFKA-17056: --- This looks interesting ! gentle ping [~chia7712] , if you are not working on this one, may I take it over ? > Convert producer state metadata schemas to use generated protocol > - > > Key: KAFKA-17056 > URL: https://issues.apache.org/jira/browse/KAFKA-17056 > Project: Kafka > Issue Type: Improvement >Reporter: Chia-Ping Tsai >Assignee: Kuan Po Tseng >Priority: Minor > > This is similar to KAFKA-10497 and KAFKA-10736 > related code: > https://github.com/apache/kafka/blob/33f5995ec379f0d18c6981106838c605ee94be7f/storage/src/main/java/org/apache/kafka/storage/internals/log/ProducerStateManager.java#L94 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-17056) Convert producer state metadata schemas to use generated protocol
[ https://issues.apache.org/jira/browse/KAFKA-17056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kuan Po Tseng reassigned KAFKA-17056: - Assignee: Kuan Po Tseng (was: Chia-Ping Tsai) > Convert producer state metadata schemas to use generated protocol > - > > Key: KAFKA-17056 > URL: https://issues.apache.org/jira/browse/KAFKA-17056 > Project: Kafka > Issue Type: Improvement >Reporter: Chia-Ping Tsai >Assignee: Kuan Po Tseng >Priority: Minor > > This is similar to KAFKA-10497 and KAFKA-10736 > related code: > https://github.com/apache/kafka/blob/33f5995ec379f0d18c6981106838c605ee94be7f/storage/src/main/java/org/apache/kafka/storage/internals/log/ProducerStateManager.java#L94 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-17038) KIP-919 supports for `alterPartitionReassignments` and `listPartitionReassignments`
[ https://issues.apache.org/jira/browse/KAFKA-17038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17859969#comment-17859969 ] Kuan Po Tseng commented on KAFKA-17038: --- Hi [~chia7712] , If you are not currently working on this issue, I am willing to take it over. Many thanks ! > KIP-919 supports for `alterPartitionReassignments` and > `listPartitionReassignments` > --- > > Key: KAFKA-17038 > URL: https://issues.apache.org/jira/browse/KAFKA-17038 > Project: Kafka > Issue Type: Sub-task >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Minor > > as title -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-17035) Add debug log to retention cleanupLogs method to help troubleshoot issues
[ https://issues.apache.org/jira/browse/KAFKA-17035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kuan Po Tseng reassigned KAFKA-17035: - Assignee: Kuan Po Tseng > Add debug log to retention cleanupLogs method to help troubleshoot issues > - > > Key: KAFKA-17035 > URL: https://issues.apache.org/jira/browse/KAFKA-17035 > Project: Kafka > Issue Type: Improvement >Reporter: Luke Chen >Assignee: Kuan Po Tseng >Priority: Major > > Sometimes, there will be some log segments leftover without getting deleted > as configured. One example is, the config is set to retention.ms = 7 days, > but there are some log segments stay in storage for more than 7 days. In this > case, it's difficult to troubleshoot if there is a bug in kafka or something > wrong in the log segment (ex: max time is wrongly set). > To help troubleshooting, we could add "trace level" or "debug level" logs > when doing retentionSize/retentionMs/logStartOffset check for each log > segment. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-17035) Add debug log to retention cleanupLogs method to help troubleshoot issues
[ https://issues.apache.org/jira/browse/KAFKA-17035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17859885#comment-17859885 ] Kuan Po Tseng commented on KAFKA-17035: --- Hi [~showuon] , I think I can help on this one, could you assign to me ? Many thanks~ > Add debug log to retention cleanupLogs method to help troubleshoot issues > - > > Key: KAFKA-17035 > URL: https://issues.apache.org/jira/browse/KAFKA-17035 > Project: Kafka > Issue Type: Improvement >Reporter: Luke Chen >Priority: Major > > Sometimes, there will be some log segments leftover without getting deleted > as configured. One example is, the config is set to retention.ms = 7 days, > but there are some log segments stay in storage for more than 7 days. In this > case, it's difficult to troubleshoot if there is a bug in kafka or something > wrong in the log segment (ex: max time is wrongly set). > To help troubleshooting, we could add "trace level" or "debug level" logs > when doing retentionSize/retentionMs/logStartOffset check for each log > segment. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-17001) Consider using another class to replace `AbstractConfig` to be class which always returns the up-to-date configs
[ https://issues.apache.org/jira/browse/KAFKA-17001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kuan Po Tseng updated KAFKA-17001: -- Summary: Consider using another class to replace `AbstractConfig` to be class which always returns the up-to-date configs (was: Consider using another class to replace `AbstractConfig` to be class which alwasy returns the up-to-date configs) > Consider using another class to replace `AbstractConfig` to be class which > always returns the up-to-date configs > > > Key: KAFKA-17001 > URL: https://issues.apache.org/jira/browse/KAFKA-17001 > Project: Kafka > Issue Type: Improvement >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Minor > > from https://github.com/apache/kafka/pull/16394#discussion_r1647321514 > We are starting to have separate config class ( i.e RemoteLogManagerConfig), > and those configs will be initialized with a AbstractConfig. By calling > `AbstractConfig' getters, those individual configs can always return the > up-to-date configs. Behind the magic behavior is the instance of > `AbstractConfig` ... yes, we use the `KafkaConfig` to construct those config > classes. We call `KafkaConfig#updateCurrentConfig` to update inner configs, > so those config classes which using `AbstractConfig` can see the latest > configs too. > However, this mechanism is not readable from `AbstractConfig`. Maybe we > should add enough docs for it. Or we can > move`KafkaConfig#updateCurrentConfig` into a new class with better naming. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (KAFKA-17024) add integration test for TransactionsCommand
[ https://issues.apache.org/jira/browse/KAFKA-17024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17859453#comment-17859453 ] Kuan Po Tseng edited comment on KAFKA-17024 at 6/23/24 5:24 AM: Hi [~chia7712] , if you are not working on this one I'm willing to solve this. Thanks ! was (Author: brandboat): Hi [~chia7712] , if you are not working on this one if you are not working on it. Thanks ! > add integration test for TransactionsCommand > > > Key: KAFKA-17024 > URL: https://issues.apache.org/jira/browse/KAFKA-17024 > Project: Kafka > Issue Type: Test >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Minor > > as title. currently we have only UT for TransactionsCommand -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-17024) add integration test for TransactionsCommand
[ https://issues.apache.org/jira/browse/KAFKA-17024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17859453#comment-17859453 ] Kuan Po Tseng commented on KAFKA-17024: --- Hi [~chia7712] , if you are not working on this one if you are not working on it. Thanks ! > add integration test for TransactionsCommand > > > Key: KAFKA-17024 > URL: https://issues.apache.org/jira/browse/KAFKA-17024 > Project: Kafka > Issue Type: Test >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Minor > > as title. currently we have only UT for TransactionsCommand -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-16830) Remove the scala version formatters support
[
https://issues.apache.org/jira/browse/KAFKA-16830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kuan Po Tseng reassigned KAFKA-16830:
-
Assignee: Kuan Po Tseng (was: Ksolves)
> Remove the scala version formatters support
> ---
>
> Key: KAFKA-16830
> URL: https://issues.apache.org/jira/browse/KAFKA-16830
> Project: Kafka
> Issue Type: Improvement
>Reporter: Chia-Ping Tsai
>Assignee: Kuan Po Tseng
>Priority: Minor
> Fix For: 4.0.0
>
>
> https://github.com/apache/kafka/blob/trunk/tools/src/main/java/org/apache/kafka/tools/consumer/ConsoleConsumerOptions.java#L353
>
> {code:java}
> private static String convertDeprecatedClass(String className) {
> switch (className) {
> case "kafka.tools.DefaultMessageFormatter":
> System.err.println("WARNING:
> kafka.tools.DefaultMessageFormatter is deprecated and will be removed in the
> next major release. " +
> "Please use
> org.apache.kafka.tools.consumer.DefaultMessageFormatter instead");
> return DefaultMessageFormatter.class.getName();
> case "kafka.tools.LoggingMessageFormatter":
> System.err.println("WARNING:
> kafka.tools.LoggingMessageFormatter is deprecated and will be removed in the
> next major release. " +
> "Please use
> org.apache.kafka.tools.consumer.LoggingMessageFormatter instead");
> return LoggingMessageFormatter.class.getName();
> case "kafka.tools.NoOpMessageFormatter":
> System.err.println("WARNING: kafka.tools.NoOpMessageFormatter
> is deprecated and will be removed in the next major release. " +
> "Please use
> org.apache.kafka.tools.consumer.NoOpMessageFormatter instead");
> return NoOpMessageFormatter.class.getName();
> default:
> return className;
> }
> }
> {code}
> Those deprecated formatters "strings" should be removed from 4.0.0
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-16830) Remove the scala version formatters support
[
https://issues.apache.org/jira/browse/KAFKA-16830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17856809#comment-17856809
]
Kuan Po Tseng commented on KAFKA-16830:
---
Hi [~ksolves.kafka] , yes, I'm working on this one.
> Remove the scala version formatters support
> ---
>
> Key: KAFKA-16830
> URL: https://issues.apache.org/jira/browse/KAFKA-16830
> Project: Kafka
> Issue Type: Improvement
>Reporter: Chia-Ping Tsai
>Assignee: Ksolves
>Priority: Minor
> Fix For: 4.0.0
>
>
> https://github.com/apache/kafka/blob/trunk/tools/src/main/java/org/apache/kafka/tools/consumer/ConsoleConsumerOptions.java#L353
>
> {code:java}
> private static String convertDeprecatedClass(String className) {
> switch (className) {
> case "kafka.tools.DefaultMessageFormatter":
> System.err.println("WARNING:
> kafka.tools.DefaultMessageFormatter is deprecated and will be removed in the
> next major release. " +
> "Please use
> org.apache.kafka.tools.consumer.DefaultMessageFormatter instead");
> return DefaultMessageFormatter.class.getName();
> case "kafka.tools.LoggingMessageFormatter":
> System.err.println("WARNING:
> kafka.tools.LoggingMessageFormatter is deprecated and will be removed in the
> next major release. " +
> "Please use
> org.apache.kafka.tools.consumer.LoggingMessageFormatter instead");
> return LoggingMessageFormatter.class.getName();
> case "kafka.tools.NoOpMessageFormatter":
> System.err.println("WARNING: kafka.tools.NoOpMessageFormatter
> is deprecated and will be removed in the next major release. " +
> "Please use
> org.apache.kafka.tools.consumer.NoOpMessageFormatter instead");
> return NoOpMessageFormatter.class.getName();
> default:
> return className;
> }
> }
> {code}
> Those deprecated formatters "strings" should be removed from 4.0.0
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (KAFKA-16900) kafka-producer-perf-test reports error when using transaction.
[ https://issues.apache.org/jira/browse/KAFKA-16900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kuan Po Tseng reassigned KAFKA-16900: - Assignee: Kuan Po Tseng > kafka-producer-perf-test reports error when using transaction. > -- > > Key: KAFKA-16900 > URL: https://issues.apache.org/jira/browse/KAFKA-16900 > Project: Kafka > Issue Type: Bug > Components: producer >Reporter: Chen He >Assignee: Kuan Po Tseng >Priority: Minor > Labels: perf-test > > [https://lists.apache.org/thread/dmrbx8kzv2w5t1v0xjvyjbp5y23omlq8] > encounter the same issue as mentioned above. > Did not found the 2.13 version in affects versions so mark it as the most > latest it provided. 2.9. Please feel free to change if possible. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16900) kafka-producer-perf-test reports error when using transaction.
[ https://issues.apache.org/jira/browse/KAFKA-16900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17856750#comment-17856750 ] Kuan Po Tseng commented on KAFKA-16900: --- Agree with [~chia7712] , the behavior of transactionsEnabled is weird... I'll start fix this, thanks > kafka-producer-perf-test reports error when using transaction. > -- > > Key: KAFKA-16900 > URL: https://issues.apache.org/jira/browse/KAFKA-16900 > Project: Kafka > Issue Type: Bug > Components: producer >Reporter: Chen He >Priority: Minor > Labels: perf-test > > [https://lists.apache.org/thread/dmrbx8kzv2w5t1v0xjvyjbp5y23omlq8] > encounter the same issue as mentioned above. > Did not found the 2.13 version in affects versions so mark it as the most > latest it provided. 2.9. Please feel free to change if possible. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16830) Remove the scala version formatters support
[
https://issues.apache.org/jira/browse/KAFKA-16830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17856734#comment-17856734
]
Kuan Po Tseng commented on KAFKA-16830:
---
hi [~ksolves.kafka] , this Jira is a follow up of
https://issues.apache.org/jira/browse/KAFKA-16795, and we are going to remove
[https://github.com/apache/kafka/blob/9b5b434e2a6b2d5290ea403fc02859b1c523d8aa/tools/src/main/java/org/apache/kafka/tools/consumer/ConsoleConsumerOptions.java#L353]
this line in 4.0.0
> Remove the scala version formatters support
> ---
>
> Key: KAFKA-16830
> URL: https://issues.apache.org/jira/browse/KAFKA-16830
> Project: Kafka
> Issue Type: Improvement
>Reporter: Chia-Ping Tsai
>Assignee: Kuan Po Tseng
>Priority: Minor
> Fix For: 4.0.0
>
>
> https://github.com/apache/kafka/blob/trunk/tools/src/main/java/org/apache/kafka/tools/consumer/ConsoleConsumerOptions.java#L353
>
> {code:java}
> private static String convertDeprecatedClass(String className) {
> switch (className) {
> case "kafka.tools.DefaultMessageFormatter":
> System.err.println("WARNING:
> kafka.tools.DefaultMessageFormatter is deprecated and will be removed in the
> next major release. " +
> "Please use
> org.apache.kafka.tools.consumer.DefaultMessageFormatter instead");
> return DefaultMessageFormatter.class.getName();
> case "kafka.tools.LoggingMessageFormatter":
> System.err.println("WARNING:
> kafka.tools.LoggingMessageFormatter is deprecated and will be removed in the
> next major release. " +
> "Please use
> org.apache.kafka.tools.consumer.LoggingMessageFormatter instead");
> return LoggingMessageFormatter.class.getName();
> case "kafka.tools.NoOpMessageFormatter":
> System.err.println("WARNING: kafka.tools.NoOpMessageFormatter
> is deprecated and will be removed in the next major release. " +
> "Please use
> org.apache.kafka.tools.consumer.NoOpMessageFormatter instead");
> return NoOpMessageFormatter.class.getName();
> default:
> return className;
> }
> }
> {code}
> Those deprecated formatters "strings" should be removed from 4.0.0
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-16998) Fix warnings in our Github actions
[ https://issues.apache.org/jira/browse/KAFKA-16998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17856552#comment-17856552 ] Kuan Po Tseng commented on KAFKA-16998: --- I can help on this one ! :) > Fix warnings in our Github actions > -- > > Key: KAFKA-16998 > URL: https://issues.apache.org/jira/browse/KAFKA-16998 > Project: Kafka > Issue Type: Task > Components: build >Reporter: Mickael Maison >Priority: Major > > Most of our Github actions produce warnings, see > [https://github.com/apache/kafka/actions/runs/9572915509|https://github.com/apache/kafka/actions/runs/9572915509.] > for example. > It looks like we need to bump the version we use for actions/checkout, > actions/setup-python, actions/upload-artifact to v4. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16975) The error message of creating `__cluster_metadata` should NOT be "Authorization failed"
[ https://issues.apache.org/jira/browse/KAFKA-16975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1780#comment-1780 ] Kuan Po Tseng commented on KAFKA-16975: --- +1 on this, perhaps we can should set `Errors.INVALID_REQUEST.code` and refine the error message. [~chia7712] , I'm willing to solve this if you are not working on this one, thanks ! > The error message of creating `__cluster_metadata` should NOT be > "Authorization failed" > > > Key: KAFKA-16975 > URL: https://issues.apache.org/jira/browse/KAFKA-16975 > Project: Kafka > Issue Type: Improvement >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Minor > > The error message of creating "__cluster_metadata" by admin is "Authorization > failed.". That could confuse users that it implies you "can" create it in > using root. However, the fact is that we DISALLOW users to create it as a > regular topic even though you are the boss :) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-16974) KRaft support in SslAdminIntegrationTest
[ https://issues.apache.org/jira/browse/KAFKA-16974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kuan Po Tseng reassigned KAFKA-16974: - Assignee: Kuan Po Tseng > KRaft support in SslAdminIntegrationTest > > > Key: KAFKA-16974 > URL: https://issues.apache.org/jira/browse/KAFKA-16974 > Project: Kafka > Issue Type: Task > Components: core >Reporter: Mickael Maison >Assignee: Kuan Po Tseng >Priority: Major > Labels: kraft-test > > This class needs to be updated to support KRaft -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16974) KRaft support in SslAdminIntegrationTest
[ https://issues.apache.org/jira/browse/KAFKA-16974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17855541#comment-17855541 ] Kuan Po Tseng commented on KAFKA-16974: --- gentle ping [~mimaison] , may I take over this issue ? I'm willing to do this ! > KRaft support in SslAdminIntegrationTest > > > Key: KAFKA-16974 > URL: https://issues.apache.org/jira/browse/KAFKA-16974 > Project: Kafka > Issue Type: Task > Components: core >Reporter: Mickael Maison >Priority: Major > Labels: kraft-test > > This class needs to be updated to support KRaft -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16942) Use ConcurrentHashMap in RecordAccumulator#nodeStats
[ https://issues.apache.org/jira/browse/KAFKA-16942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kuan Po Tseng resolved KAFKA-16942. --- Resolution: Won't Do The performance in get method is not obvious between ConcurrentHashMap and CopyOnWriteMap. And doing this breaks the code consistency (some place use CopyOnWriteMap while some are not). This seems not bring too much benefit. > Use ConcurrentHashMap in RecordAccumulator#nodeStats > > > Key: KAFKA-16942 > URL: https://issues.apache.org/jira/browse/KAFKA-16942 > Project: Kafka > Issue Type: Improvement > Components: clients >Reporter: Kuan Po Tseng >Assignee: Kuan Po Tseng >Priority: Major > > per discussed in > [https://github.com/apache/kafka/pull/16231#discussion_r1635345881] > Through the ConcurrentMapBenchmark, we observed that in scenarios where write > operations (i.e., computeIfAbsent) constitute 10%, the get performance of > CopyOnWriteMap is lower compared to ConcurrentHashMap. However, when > iterating over entrySet and values, CopyOnWriteMap performs better than > ConcurrentHashMap. > In RecordAccumulator#nodeStats, the computeIfAbsent method is rarely > triggered, and we only use the get method to read data. Therefore, switching > to ConcurrentHashMap would gain better performance. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16942) Use ConcurrentHashMap in RecordAccumulator#nodeStats
Kuan Po Tseng created KAFKA-16942: - Summary: Use ConcurrentHashMap in RecordAccumulator#nodeStats Key: KAFKA-16942 URL: https://issues.apache.org/jira/browse/KAFKA-16942 Project: Kafka Issue Type: Improvement Components: clients Reporter: Kuan Po Tseng Assignee: Kuan Po Tseng per discussed in [https://github.com/apache/kafka/pull/16231#discussion_r1635345881] Through the ConcurrentMapBenchmark, we observed that in scenarios where write operations (i.e., computeIfAbsent) constitute 10%, the get performance of CopyOnWriteMap is lower compared to ConcurrentHashMap. However, when iterating over entrySet and values, CopyOnWriteMap performs better than ConcurrentHashMap. In RecordAccumulator#nodeStats, the computeIfAbsent method is rarely triggered, and we only use the get method to read data. Therefore, switching to ConcurrentHashMap would gain better performance. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16909) Refactor GroupCoordinatorConfig with AbstractConfig
[ https://issues.apache.org/jira/browse/KAFKA-16909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17852861#comment-17852861 ] Kuan Po Tseng commented on KAFKA-16909: --- Gentle ping [~chia7712] , may I take over this one ? > Refactor GroupCoordinatorConfig with AbstractConfig > --- > > Key: KAFKA-16909 > URL: https://issues.apache.org/jira/browse/KAFKA-16909 > Project: Kafka > Issue Type: Improvement >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Minor > > This is similar to KAFKA-16884 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (KAFKA-16888) Fix failed StorageToolTest.testFormatSucceedsIfAllDirectoriesAreAvailable and StorageToolTest.testFormatEmptyDirectory
[ https://issues.apache.org/jira/browse/KAFKA-16888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17851931#comment-17851931 ] Kuan Po Tseng edited comment on KAFKA-16888 at 6/4/24 7:58 AM: --- filed https://github.com/apache/kafka/pull/16186 please take a look, thanks ! was (Author: brandboat): filed [https://github.com/apache/kafka/pull/16186|https://github.com/apache/kafka/pull/16186,] please take a look, thanks ! > Fix failed StorageToolTest.testFormatSucceedsIfAllDirectoriesAreAvailable and > StorageToolTest.testFormatEmptyDirectory > -- > > Key: KAFKA-16888 > URL: https://issues.apache.org/jira/browse/KAFKA-16888 > Project: Kafka > Issue Type: Bug >Reporter: Chia-Ping Tsai >Assignee: Kuan Po Tseng >Priority: Minor > > The commit > (https://github.com/apache/kafka/commit/459da4795a511f6933e940fcf105a824bd9e589c#diff-4bacfdbf0e63a4d5f3deb1a0d39037a18510ac24ee5ec276fe70bc818ba4d209L505) > added new string to `stream`, so the test case will fail due to extra output. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (KAFKA-16888) Fix failed StorageToolTest.testFormatSucceedsIfAllDirectoriesAreAvailable and StorageToolTest.testFormatEmptyDirectory
[ https://issues.apache.org/jira/browse/KAFKA-16888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17851931#comment-17851931 ] Kuan Po Tseng edited comment on KAFKA-16888 at 6/4/24 7:57 AM: --- filed [https://github.com/apache/kafka/pull/16186|https://github.com/apache/kafka/pull/16186,] please take a look, thanks ! was (Author: brandboat): filed [https://github.com/apache/kafka/pull/16186,] please take a look, thanks ! > Fix failed StorageToolTest.testFormatSucceedsIfAllDirectoriesAreAvailable and > StorageToolTest.testFormatEmptyDirectory > -- > > Key: KAFKA-16888 > URL: https://issues.apache.org/jira/browse/KAFKA-16888 > Project: Kafka > Issue Type: Bug >Reporter: Chia-Ping Tsai >Assignee: Kuan Po Tseng >Priority: Minor > > The commit > (https://github.com/apache/kafka/commit/459da4795a511f6933e940fcf105a824bd9e589c#diff-4bacfdbf0e63a4d5f3deb1a0d39037a18510ac24ee5ec276fe70bc818ba4d209L505) > added new string to `stream`, so the test case will fail due to extra output. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16888) Fix failed StorageToolTest.testFormatSucceedsIfAllDirectoriesAreAvailable and StorageToolTest.testFormatEmptyDirectory
[ https://issues.apache.org/jira/browse/KAFKA-16888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17851931#comment-17851931 ] Kuan Po Tseng commented on KAFKA-16888: --- filed [https://github.com/apache/kafka/pull/16186,] please take a look, thanks ! > Fix failed StorageToolTest.testFormatSucceedsIfAllDirectoriesAreAvailable and > StorageToolTest.testFormatEmptyDirectory > -- > > Key: KAFKA-16888 > URL: https://issues.apache.org/jira/browse/KAFKA-16888 > Project: Kafka > Issue Type: Bug >Reporter: Chia-Ping Tsai >Assignee: Kuan Po Tseng >Priority: Minor > > The commit > (https://github.com/apache/kafka/commit/459da4795a511f6933e940fcf105a824bd9e589c#diff-4bacfdbf0e63a4d5f3deb1a0d39037a18510ac24ee5ec276fe70bc818ba4d209L505) > added new string to `stream`, so the test case will fail due to extra output. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16888) Fix failed StorageToolTest.testFormatSucceedsIfAllDirectoriesAreAvailable and StorageToolTest.testFormatEmptyDirectory
[ https://issues.apache.org/jira/browse/KAFKA-16888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17851920#comment-17851920 ] Kuan Po Tseng commented on KAFKA-16888: --- Sure, I'll fix this asap > Fix failed StorageToolTest.testFormatSucceedsIfAllDirectoriesAreAvailable and > StorageToolTest.testFormatEmptyDirectory > -- > > Key: KAFKA-16888 > URL: https://issues.apache.org/jira/browse/KAFKA-16888 > Project: Kafka > Issue Type: Bug >Reporter: Chia-Ping Tsai >Assignee: Kuan Po Tseng >Priority: Minor > > The commit > (https://github.com/apache/kafka/commit/459da4795a511f6933e940fcf105a824bd9e589c#diff-4bacfdbf0e63a4d5f3deb1a0d39037a18510ac24ee5ec276fe70bc818ba4d209L505) > added new string to `stream`, so the test case will fail due to extra output. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-16888) Fix failed StorageToolTest.testFormatSucceedsIfAllDirectoriesAreAvailable
[ https://issues.apache.org/jira/browse/KAFKA-16888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kuan Po Tseng reassigned KAFKA-16888: - Assignee: Kuan Po Tseng (was: Chia-Ping Tsai) > Fix failed StorageToolTest.testFormatSucceedsIfAllDirectoriesAreAvailable > - > > Key: KAFKA-16888 > URL: https://issues.apache.org/jira/browse/KAFKA-16888 > Project: Kafka > Issue Type: Bug >Reporter: Chia-Ping Tsai >Assignee: Kuan Po Tseng >Priority: Minor > > The commit > (https://github.com/apache/kafka/commit/459da4795a511f6933e940fcf105a824bd9e589c#diff-4bacfdbf0e63a4d5f3deb1a0d39037a18510ac24ee5ec276fe70bc818ba4d209L505) > added new string to `stream`, so the test case will fail due to extra output. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16877) Migrate RemoteLogSegmentLifecycleTest to use new test infra
[ https://issues.apache.org/jira/browse/KAFKA-16877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17851405#comment-17851405 ] Kuan Po Tseng commented on KAFKA-16877: --- I'm willing to take over this. Many thanks. > Migrate RemoteLogSegmentLifecycleTest to use new test infra > --- > > Key: KAFKA-16877 > URL: https://issues.apache.org/jira/browse/KAFKA-16877 > Project: Kafka > Issue Type: Test >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Minor > > https://github.com/apache/kafka/blob/2c82ecd67f2f6b412f625e8efc1457e7fb7f74dd/storage/src/test/java/org/apache/kafka/server/log/remote/metadata/storage/RemoteLogSegmentLifecycleTest.java#L432 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-16814) KRaft broker cannot startup when `partition.metadata` is missing
[
https://issues.apache.org/jira/browse/KAFKA-16814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kuan Po Tseng reassigned KAFKA-16814:
-
Assignee: Kuan Po Tseng
> KRaft broker cannot startup when `partition.metadata` is missing
>
>
> Key: KAFKA-16814
> URL: https://issues.apache.org/jira/browse/KAFKA-16814
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 3.7.0
>Reporter: Luke Chen
>Assignee: Kuan Po Tseng
>Priority: Major
>
> When starting up kafka logManager, we'll check stray replicas to avoid some
> corner cases. But this check might cause broker unable to startup if
> `partition.metadata` is missing because when startup kafka, we load log from
> file, and the topicId of the log is coming from `partition.metadata` file.
> So, if `partition.metadata` is missing, the topicId will be None, and the
> `LogManager#isStrayKraftReplica` will fail with no topicID error.
> The `partition.metadata` missing could be some storage failure, or another
> possible path is unclean shutdown after topic is created in the replica, but
> before data is flushed into `partition.metadata` file. This is possible
> because we do the flush in async way
> [here|https://github.com/apache/kafka/blob/5552f5c26df4eb07b2d6ee218e4a29e4ca790d5c/core/src/main/scala/kafka/log/UnifiedLog.scala#L229].
>
>
> {code:java}
> ERROR Encountered fatal fault: Error starting LogManager
> (org.apache.kafka.server.fault.ProcessTerminatingFaultHandler)
> java.lang.RuntimeException: The log dir
> Log(dir=/tmp/kraft-broker-logs/quickstart-events-0, topic=quickstart-events,
> partition=0, highWatermark=0, lastStableOffset=0, logStartOffset=0,
> logEndOffset=0) does not have a topic ID, which is not allowed when running
> in KRaft mode.
> at
> kafka.log.LogManager$.$anonfun$isStrayKraftReplica$1(LogManager.scala:1609)
> at scala.Option.getOrElse(Option.scala:201)
> at kafka.log.LogManager$.isStrayKraftReplica(LogManager.scala:1608)
> at
> kafka.server.metadata.BrokerMetadataPublisher.$anonfun$initializeManagers$1(BrokerMetadataPublisher.scala:294)
> at
> kafka.server.metadata.BrokerMetadataPublisher.$anonfun$initializeManagers$1$adapted(BrokerMetadataPublisher.scala:294)
> at kafka.log.LogManager.loadLog(LogManager.scala:359)
> at kafka.log.LogManager.$anonfun$loadLogs$15(LogManager.scala:493)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:577)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
> at java.base/java.lang.Thread.run(Thread.java:1623) {code}
>
> Because if we don't do the isStrayKraftReplica check, the topicID and the
> `partition.metadata` will get recovered after getting topic partition update
> and becoming leader or follower later. I'm proposing we skip the
> `isStrayKraftReplica` check if topicID is None, instead of throwing exception
> to terminate the kafka. `isStrayKraftReplica` check is just for a corner case
> only, it should be fine IMO.
>
>
> === update ===
> Checked KAFKA-14616 and KAFKA-15605, our purpose of finding strayReplicas and
> delete them is because the replica should be deleted, but left in the log
> dir. So, if we have a replica that doesn't have topicID (due to
> `partition.metadata` is missing), then we cannot identify if this is a stray
> replica or not. In this case, we can do:
> # Delete it
> # Ignore it
> For (1), the impact is, if this is not a stray replica, and the
> replication-factor only has 1, then the data might be moved to another
> "xxx-stray" dir, and the partition becomes empty.
> For (2), the impact is, if this is a stray replica and we didn't delete it,
> it might cause partition dir is not created as in KAFKA-15605 or KAFKA-14616.
> As the investigation above, this `partition.metadata` missing issue is mostly
> because the async `partition.metadata` when creating a topic. Later, before
> any data append into log, we must make sure partition metadata file is
> written to the log dir
> [here|https://github.com/apache/kafka/blob/5552f5c26df4eb07b2d6ee218e4a29e4ca790d5c/core/src/main/scala/kafka/log/UnifiedLog.scala#L772-L774].
> So, it should be fine if we delete it since the topic should be empty.
> In short, when finding a log without topicID, we should treat it as a stray
> log and then delete it.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-16814) KRaft broker cannot startup when `partition.metadata` is missing
[
https://issues.apache.org/jira/browse/KAFKA-16814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17849917#comment-17849917
]
Kuan Po Tseng commented on KAFKA-16814:
---
gentle ping [~showuon] , I'm willing to solve this issue. Will file a PR soon,
many thanks~
> KRaft broker cannot startup when `partition.metadata` is missing
>
>
> Key: KAFKA-16814
> URL: https://issues.apache.org/jira/browse/KAFKA-16814
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 3.7.0
>Reporter: Luke Chen
>Priority: Major
>
> When starting up kafka logManager, we'll check stray replicas to avoid some
> corner cases. But this check might cause broker unable to startup if
> `partition.metadata` is missing because when startup kafka, we load log from
> file, and the topicId of the log is coming from `partition.metadata` file.
> So, if `partition.metadata` is missing, the topicId will be None, and the
> `LogManager#isStrayKraftReplica` will fail with no topicID error.
> The `partition.metadata` missing could be some storage failure, or another
> possible path is unclean shutdown after topic is created in the replica, but
> before data is flushed into `partition.metadata` file. This is possible
> because we do the flush in async way
> [here|https://github.com/apache/kafka/blob/5552f5c26df4eb07b2d6ee218e4a29e4ca790d5c/core/src/main/scala/kafka/log/UnifiedLog.scala#L229].
>
>
> {code:java}
> ERROR Encountered fatal fault: Error starting LogManager
> (org.apache.kafka.server.fault.ProcessTerminatingFaultHandler)
> java.lang.RuntimeException: The log dir
> Log(dir=/tmp/kraft-broker-logs/quickstart-events-0, topic=quickstart-events,
> partition=0, highWatermark=0, lastStableOffset=0, logStartOffset=0,
> logEndOffset=0) does not have a topic ID, which is not allowed when running
> in KRaft mode.
> at
> kafka.log.LogManager$.$anonfun$isStrayKraftReplica$1(LogManager.scala:1609)
> at scala.Option.getOrElse(Option.scala:201)
> at kafka.log.LogManager$.isStrayKraftReplica(LogManager.scala:1608)
> at
> kafka.server.metadata.BrokerMetadataPublisher.$anonfun$initializeManagers$1(BrokerMetadataPublisher.scala:294)
> at
> kafka.server.metadata.BrokerMetadataPublisher.$anonfun$initializeManagers$1$adapted(BrokerMetadataPublisher.scala:294)
> at kafka.log.LogManager.loadLog(LogManager.scala:359)
> at kafka.log.LogManager.$anonfun$loadLogs$15(LogManager.scala:493)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:577)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
> at java.base/java.lang.Thread.run(Thread.java:1623) {code}
>
> Because if we don't do the isStrayKraftReplica check, the topicID and the
> `partition.metadata` will get recovered after getting topic partition update
> and becoming leader or follower later. I'm proposing we skip the
> `isStrayKraftReplica` check if topicID is None, instead of throwing exception
> to terminate the kafka. `isStrayKraftReplica` check is just for a corner case
> only, it should be fine IMO.
>
>
> === update ===
> Checked KAFKA-14616 and KAFKA-15605, our purpose of finding strayReplicas and
> delete them is because the replica should be deleted, but left in the log
> dir. So, if we have a replica that doesn't have topicID (due to
> `partition.metadata` is missing), then we cannot identify if this is a stray
> replica or not. In this case, we can do:
> # Delete it
> # Ignore it
> For (1), the impact is, if this is not a stray replica, and the
> replication-factor only has 1, then the data might be moved to another
> "xxx-stray" dir, and the partition becomes empty.
> For (2), the impact is, if this is a stray replica and we didn't delete it,
> it might cause partition dir is not created as in KAFKA-15605 or KAFKA-14616.
> As the investigation above, this `partition.metadata` missing issue is mostly
> because the async `partition.metadata` when creating a topic. Later, before
> any data append into log, we must make sure partition metadata file is
> written to the log dir
> [here|https://github.com/apache/kafka/blob/5552f5c26df4eb07b2d6ee218e4a29e4ca790d5c/core/src/main/scala/kafka/log/UnifiedLog.scala#L772-L774].
> So, it should be fine if we delete it since the topic should be empty.
> In short, when finding a log without topicID, we should treat it as a stray
> log and then delete it.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-12708) Rewrite org.apache.kafka.test.Microbenchmarks by JMH
[ https://issues.apache.org/jira/browse/KAFKA-12708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17849780#comment-17849780 ] Kuan Po Tseng commented on KAFKA-12708: --- gentle ping [~Geordie] , are you still working on this one ? I saw the heavy works are almost done in your PR, perhaps I can rebased the pr and re-test it. > Rewrite org.apache.kafka.test.Microbenchmarks by JMH > > > Key: KAFKA-12708 > URL: https://issues.apache.org/jira/browse/KAFKA-12708 > Project: Kafka > Issue Type: Task >Reporter: Chia-Ping Tsai >Assignee: GeordieMai >Priority: Minor > > The benchmark code is a bit obsolete and it would be better to rewrite it by > JMH -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-16828) RackAwareTaskAssignorTest failed
[ https://issues.apache.org/jira/browse/KAFKA-16828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kuan Po Tseng reassigned KAFKA-16828: - Assignee: Kuan Po Tseng > RackAwareTaskAssignorTest failed > > > Key: KAFKA-16828 > URL: https://issues.apache.org/jira/browse/KAFKA-16828 > Project: Kafka > Issue Type: Test >Reporter: Luke Chen >Assignee: Kuan Po Tseng >Priority: Major > > Found in the latest trunk build. > It fails many tests in `RackAwareTaskAssignorTest` suite. > > https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-15951/7/#showFailuresLink -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-16795) Fix broken compatibility in kafka.tools.NoOpMessageFormatter, kafka.tools.DefaultMessageFormatter, and kafka.tools.LoggingMessageFormatter
[
https://issues.apache.org/jira/browse/KAFKA-16795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kuan Po Tseng updated KAFKA-16795:
--
Parent: KAFKA-14525
Issue Type: Sub-task (was: Bug)
> Fix broken compatibility in kafka.tools.NoOpMessageFormatter,
> kafka.tools.DefaultMessageFormatter, and kafka.tools.LoggingMessageFormatter
> --
>
> Key: KAFKA-16795
> URL: https://issues.apache.org/jira/browse/KAFKA-16795
> Project: Kafka
> Issue Type: Sub-task
>Reporter: Kuan Po Tseng
>Assignee: Kuan Po Tseng
>Priority: Major
> Fix For: 3.8.0
>
>
> [{{0bf830f}}|https://github.com/apache/kafka/commit/0bf830fc9c3915bc99b6e487e6083dabd593c5d3]
> moved NoOpMessageFormatter, DefaultMessageFormatter and
> LoggingMessageFormatter package from {{kafka.tools}} to
> {{{}org.apache.kafka.tools.consumer{}}}{{{}{}}}
> These classes could be used via cmd kafka-console-consumer.sh. We should have
> a dependency cycle before 3.8.0 comes out.
>
> {code:java}
> bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 \
> --topic streams-wordcount-output \
> --from-beginning \
> --formatter kafka.tools.DefaultMessageFormatter \
> --property print.key=true \
> --property print.value=true \
> --property
> key.deserializer=org.apache.kafka.common.serialization.StringDeserializer \
> --property
> value.deserializer=org.apache.kafka.common.serialization.LongDeserializer{code}
> The goal in this Jira is to allow user to keep using
> {{{}kafka.tools.NoOpMessageFormatter{}}},
> {{{}kafka.tools.DefaultMessageFormatter{}}}, and
> {{{}kafka.tools.LoggingMessageFormatter{}}}, but we also display warning
> messages to say those "strings" will be removed in 4.0.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KAFKA-16795) Fix broken compatibility in kafka.tools.NoOpMessageFormatter, kafka.tools.DefaultMessageFormatter, and kafka.tools.LoggingMessageFormatter
Kuan Po Tseng created KAFKA-16795:
-
Summary: Fix broken compatibility in
kafka.tools.NoOpMessageFormatter, kafka.tools.DefaultMessageFormatter, and
kafka.tools.LoggingMessageFormatter
Key: KAFKA-16795
URL: https://issues.apache.org/jira/browse/KAFKA-16795
Project: Kafka
Issue Type: Bug
Reporter: Kuan Po Tseng
Assignee: Kuan Po Tseng
Fix For: 3.8.0
[{{0bf830f}}|https://github.com/apache/kafka/commit/0bf830fc9c3915bc99b6e487e6083dabd593c5d3]
moved NoOpMessageFormatter, DefaultMessageFormatter and
LoggingMessageFormatter package from {{kafka.tools}} to
{{{}org.apache.kafka.tools.consumer{}}}{{{}{}}}
These classes could be used via cmd kafka-console-consumer.sh. We should have a
dependency cycle before 3.8.0 comes out.
{code:java}
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 \
--topic streams-wordcount-output \
--from-beginning \
--formatter kafka.tools.DefaultMessageFormatter \
--property print.key=true \
--property print.value=true \
--property
key.deserializer=org.apache.kafka.common.serialization.StringDeserializer \
--property
value.deserializer=org.apache.kafka.common.serialization.LongDeserializer{code}
The goal in this Jira is to allow user to keep using
{{{}kafka.tools.NoOpMessageFormatter{}}},
{{{}kafka.tools.DefaultMessageFormatter{}}}, and
{{{}kafka.tools.LoggingMessageFormatter{}}}, but we also display warning
messages to say those "strings" will be removed in 4.0.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KAFKA-16794) Can't open videos in streams documentation
Kuan Po Tseng created KAFKA-16794:
-
Summary: Can't open videos in streams documentation
Key: KAFKA-16794
URL: https://issues.apache.org/jira/browse/KAFKA-16794
Project: Kafka
Issue Type: Bug
Components: docs
Reporter: Kuan Po Tseng
Attachments: IMG_4445.png, image.png
Can't open videos in page [https://kafka.apache.org/documentation/streams/]
Open console in chrome browser and it shows error message:
{{Refused to frame 'https://www.youtube.com/' because it violates the following
Content Security Policy directive: "frame-src 'self'".}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-16414) Inconsistent active segment expiration behavior between retention.ms and retention.bytes
[ https://issues.apache.org/jira/browse/KAFKA-16414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17847617#comment-17847617 ] Kuan Po Tseng commented on KAFKA-16414: --- c.c. [~showuon] , any thoughts on this one ? Would be appreciate if you can give any feedback. Many thanks :) > Inconsistent active segment expiration behavior between retention.ms and > retention.bytes > > > Key: KAFKA-16414 > URL: https://issues.apache.org/jira/browse/KAFKA-16414 > Project: Kafka > Issue Type: Improvement >Affects Versions: 3.6.1 >Reporter: Kuan Po Tseng >Assignee: Kuan Po Tseng >Priority: Major > > This is a follow up issue on KAFKA-16385. > Currently, there's a difference between how retention.ms and retention.bytes > handle active segment expiration: > - retention.ms always expire active segment when max segment timestamp > matches the condition. > - retention.bytes only expire active segment when retention.bytes is > configured to zero. > The behavior should be either rotate active segments for both retention > configurations or none at all. > For more details, see > https://issues.apache.org/jira/browse/KAFKA-16385?focusedCommentId=17829682=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17829682 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16414) Inconsistent active segment expiration behavior between retention.ms and retention.bytes
[ https://issues.apache.org/jira/browse/KAFKA-16414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17847616#comment-17847616 ] Kuan Po Tseng commented on KAFKA-16414: --- I'm still waiting for other Kafka PMC members/committers to agree on this. So far, it seems that only [~chia7712] give a +1 on this. > Inconsistent active segment expiration behavior between retention.ms and > retention.bytes > > > Key: KAFKA-16414 > URL: https://issues.apache.org/jira/browse/KAFKA-16414 > Project: Kafka > Issue Type: Improvement >Affects Versions: 3.6.1 >Reporter: Kuan Po Tseng >Assignee: Kuan Po Tseng >Priority: Major > > This is a follow up issue on KAFKA-16385. > Currently, there's a difference between how retention.ms and retention.bytes > handle active segment expiration: > - retention.ms always expire active segment when max segment timestamp > matches the condition. > - retention.bytes only expire active segment when retention.bytes is > configured to zero. > The behavior should be either rotate active segments for both retention > configurations or none at all. > For more details, see > https://issues.apache.org/jira/browse/KAFKA-16385?focusedCommentId=17829682=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17829682 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16785) Migrate TopicBasedRemoteLogMetadataManagerRestartTest to new test infra
[ https://issues.apache.org/jira/browse/KAFKA-16785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17847019#comment-17847019 ] Kuan Po Tseng commented on KAFKA-16785: --- May I take over this Jira ? > Migrate TopicBasedRemoteLogMetadataManagerRestartTest to new test infra > --- > > Key: KAFKA-16785 > URL: https://issues.apache.org/jira/browse/KAFKA-16785 > Project: Kafka > Issue Type: Test >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Minor > Labels: storage_test > > as title -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16595) Introduce ClusterTemplate in ClusterTests
[ https://issues.apache.org/jira/browse/KAFKA-16595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kuan Po Tseng resolved KAFKA-16595. --- Resolution: Won't Do As discussed in [https://github.com/apache/kafka/pull/15899#discussion_r1594890663.] We have other ways to simplify the test cases. > Introduce ClusterTemplate in ClusterTests > - > > Key: KAFKA-16595 > URL: https://issues.apache.org/jira/browse/KAFKA-16595 > Project: Kafka > Issue Type: Improvement >Reporter: Kuan Po Tseng >Assignee: Kuan Po Tseng >Priority: Major > > discussed in https://github.com/apache/kafka/pull/15761#discussion_r1573850549 > Currently we can't apply any template in ClusterTests, thus we have to write > down all ClusterConfigProperty in each ClusterTest inside ClusterTests. And > that could leave bunch of duplicate code. We need to find a way to reduce the > duplicate code. Introduce template in ClusterTests could be a solution. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-16595) Introduce ClusterTemplate in ClusterTests
[ https://issues.apache.org/jira/browse/KAFKA-16595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kuan Po Tseng updated KAFKA-16595: -- Summary: Introduce ClusterTemplate in ClusterTests (was: Introduce template in ClusterTests) > Introduce ClusterTemplate in ClusterTests > - > > Key: KAFKA-16595 > URL: https://issues.apache.org/jira/browse/KAFKA-16595 > Project: Kafka > Issue Type: Improvement >Reporter: Kuan Po Tseng >Assignee: Kuan Po Tseng >Priority: Major > > discussed in https://github.com/apache/kafka/pull/15761#discussion_r1573850549 > Currently we can't apply any template in ClusterTests, thus we have to write > down all ClusterConfigProperty in each ClusterTest inside ClusterTests. And > that could leave bunch of duplicate code. We need to find a way to reduce the > duplicate code. Introduce template in ClusterTests could be a solution. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16660) reduce the check interval to speedup DelegationTokenRequestsTest
[ https://issues.apache.org/jira/browse/KAFKA-16660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17843171#comment-17843171 ] Kuan Po Tseng commented on KAFKA-16660: --- I'm willing to work on this item ! May I take over this ? :) > reduce the check interval to speedup DelegationTokenRequestsTest > > > Key: KAFKA-16660 > URL: https://issues.apache.org/jira/browse/KAFKA-16660 > Project: Kafka > Issue Type: Improvement >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Minor > > the check interval is 1 minute > (https://github.com/apache/kafka/blob/trunk/core/src/test/scala/unit/kafka/server/DelegationTokenRequestsTest.scala#L49), > and `DelegationTokenRequestsTest` waits 2 minutes before running the check > (https://github.com/apache/kafka/blob/trunk/core/src/test/scala/unit/kafka/server/DelegationTokenRequestsTest.scala#L159) > ... > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16650) add integration test for Admin#abortTransaction
[ https://issues.apache.org/jira/browse/KAFKA-16650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17842542#comment-17842542 ] Kuan Po Tseng commented on KAFKA-16650: --- May I take over this issue ? :) > add integration test for Admin#abortTransaction > --- > > Key: KAFKA-16650 > URL: https://issues.apache.org/jira/browse/KAFKA-16650 > Project: Kafka > Issue Type: Improvement >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Minor > > It seems there are only few unit tests. We should add IT includeing zk, > kraft, and new group coordinator for it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16627) Remove ClusterConfig parameter in BeforeEach and AfterEach
Kuan Po Tseng created KAFKA-16627: - Summary: Remove ClusterConfig parameter in BeforeEach and AfterEach Key: KAFKA-16627 URL: https://issues.apache.org/jira/browse/KAFKA-16627 Project: Kafka Issue Type: Improvement Reporter: Kuan Po Tseng Assignee: Kuan Po Tseng In the past we modify configs like server broker properties by modifying the ClusterConfig reference passed to BeforeEach and AfterEach based on the requirements of the tests. While after KAFKA-16560, the ClusterConfig become immutable, modify the ClusterConfig reference no longer reflects any changes to the test cluster. Then pass ClusterConfig to BeforeEach and AfterEach become redundant. We should remove this behavior. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16595) Introduce template in ClusterTests
Kuan Po Tseng created KAFKA-16595: - Summary: Introduce template in ClusterTests Key: KAFKA-16595 URL: https://issues.apache.org/jira/browse/KAFKA-16595 Project: Kafka Issue Type: Improvement Reporter: Kuan Po Tseng Assignee: Kuan Po Tseng discussed in https://github.com/apache/kafka/pull/15761#discussion_r1573850549 Currently we can't apply any template in ClusterTests, thus we have to write down all ClusterConfigProperty in each ClusterTest inside ClusterTests. And that could leave bunch of duplicate code. We need to find a way to reduce the duplicate code. Introduce template in ClusterTests could be a solution. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16560) Refactor/cleanup BrokerNode/ControllerNode/ClusterConfig
[ https://issues.apache.org/jira/browse/KAFKA-16560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17837553#comment-17837553 ] Kuan Po Tseng commented on KAFKA-16560: --- Thanks [~chia7712], in https://github.com/apache/kafka/pull/15715, currently I'm working on making ClusterConfig immutable. Like you mentioned in pr comment, there are still plenty of places like BrokerNode, ControllerNode are not immutable. We can continue the work in this JIRA. > Refactor/cleanup BrokerNode/ControllerNode/ClusterConfig > > > Key: KAFKA-16560 > URL: https://issues.apache.org/jira/browse/KAFKA-16560 > Project: Kafka > Issue Type: Improvement >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Minor > > origin discussion: > https://github.com/apache/kafka/pull/15715#discussion_r1564660916 > It seems to me this jira should address following tasks. > 1. make them immutable. We have adopted the builder pattern, so all changes > should be completed in the builder phase > 2. make all `Builder#build()` not accept any arguments. Instead, we should > add new setters for those arguments. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16552) Create an internal config to control InitialTaskDelayMs in LogManager to speed up tests
[ https://issues.apache.org/jira/browse/KAFKA-16552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17837163#comment-17837163 ] Kuan Po Tseng commented on KAFKA-16552: --- +1 on this ! > Create an internal config to control InitialTaskDelayMs in LogManager to > speed up tests > --- > > Key: KAFKA-16552 > URL: https://issues.apache.org/jira/browse/KAFKA-16552 > Project: Kafka > Issue Type: Improvement >Reporter: Luke Chen >Assignee: Luke Chen >Priority: Major > > When startup LogManager, we'll create schedule tasks like: > kafka-log-retention, kafka-recovery-point-checkpoint threads...etc > ([here|https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/log/LogManager.scala#L629]). > All of them have public configs to configure the interval, like > `log.retention.check.interval.ms`. But in addition to the scheduler interval, > there's a hard coded InitialTaskDelayMs (30 seconds) for all of them. That > might not be a problem in production env, since it'll make the kafka server > start up faster. But in test env, the 30 secs delay means if there are tests > verifying the behaviors like log retention, it'll take 30 secs up to complete > the tests. > To speed up tests, we should create an internal config (ex: > "log.initial.task.delay.ms") to control InitialTaskDelayMs in LogManager to > speed up tests. This is not intended to be used by normal users, just for > speeding up testing usage. > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16544) DescribeTopicsResult#allTopicIds and DescribeTopicsResult#allTopicNames should return null instead of throwing NPE
[
https://issues.apache.org/jira/browse/KAFKA-16544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17836797#comment-17836797
]
Kuan Po Tseng commented on KAFKA-16544:
---
gentle ping [~chia7712], are you working on this one ? Maybe I can give it a
try ! :)
> DescribeTopicsResult#allTopicIds and DescribeTopicsResult#allTopicNames
> should return null instead of throwing NPE
> --
>
> Key: KAFKA-16544
> URL: https://issues.apache.org/jira/browse/KAFKA-16544
> Project: Kafka
> Issue Type: Improvement
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Major
>
> {code:java}
> * @return A future map from topic names to descriptions which can be
> used to check
> * the status of individual description if the describe topic
> request used
> * topic names, otherwise return null, this request succeeds only
> if all the
> * topic descriptions succeed
> {code}
> According the docs, it should return null if we try to get the result
> unmatched to the request. For example, we call `allTopicNames` in passing
> `TopicIdCollection`. However, the current implementation will throw NPE
> directly
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (KAFKA-16484) Support to define per broker/controller property by ClusterConfigProperty
[ https://issues.apache.org/jira/browse/KAFKA-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17834690#comment-17834690 ] Kuan Po Tseng edited comment on KAFKA-16484 at 4/7/24 3:14 PM: --- Hello [~chia7712], Not sure if you're working on this issue, if not, I'm willing to take over this issue, many thanks ! was (Author: brandboat): Hello [~chia7712], I'm willing to take over this issue, many thanks ! > Support to define per broker/controller property by ClusterConfigProperty > - > > Key: KAFKA-16484 > URL: https://issues.apache.org/jira/browse/KAFKA-16484 > Project: Kafka > Issue Type: Test >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Major > > the property set to `ClusterConfigProperty` gets applied to all brokers, and > hence we can't have individual props for each broker to test racks. > > It seems to me we can add new field "id" to `ClusterConfigProperty` to > declare the property should be applied to specific broker (or controller). > the default value is -1 and it should be applied to all nodes. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16484) Support to define per broker/controller property by ClusterConfigProperty
[ https://issues.apache.org/jira/browse/KAFKA-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17834690#comment-17834690 ] Kuan Po Tseng commented on KAFKA-16484: --- Hello [~chia7712], I'm willing to take over this issue, many thanks ! > Support to define per broker/controller property by ClusterConfigProperty > - > > Key: KAFKA-16484 > URL: https://issues.apache.org/jira/browse/KAFKA-16484 > Project: Kafka > Issue Type: Test >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Major > > the property set to `ClusterConfigProperty` gets applied to all brokers, and > hence we can't have individual props for each broker to test racks. > > It seems to me we can add new field "id" to `ClusterConfigProperty` to > declare the property should be applied to specific broker (or controller). > the default value is -1 and it should be applied to all nodes. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16477) Detect thread leaked client-metrics-reaper in tests
Kuan Po Tseng created KAFKA-16477:
-
Summary: Detect thread leaked client-metrics-reaper in tests
Key: KAFKA-16477
URL: https://issues.apache.org/jira/browse/KAFKA-16477
Project: Kafka
Issue Type: Improvement
Reporter: Kuan Po Tseng
Assignee: Kuan Po Tseng
After profiling the kafka tests, tons of `client-metrics-reaper` thread not
cleanup after BrokerServer shutdown.
The thread {{client-metrics-reaper}} comes from
[ClientMetricsManager#expirationTimer|https://github.com/apache/kafka/blob/a2ee0855ee5e73f3a74555d52294bb4acfd28945/server/src/main/java/org/apache/kafka/server/ClientMetricsManager.java#L115],
and BrokerServer#shudown doesn't close ClientMetricsManager which let the
timer thread still runs in background.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-16473) KafkaDockerWrapper uses wrong cluster ID when formatting log dir
[
https://issues.apache.org/jira/browse/KAFKA-16473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17834001#comment-17834001
]
Kuan Po Tseng commented on KAFKA-16473:
---
> I would appreciate a review of that PR, though.
Sure, I'll do that. Thanks for your input!
> KafkaDockerWrapper uses wrong cluster ID when formatting log dir
>
>
> Key: KAFKA-16473
> URL: https://issues.apache.org/jira/browse/KAFKA-16473
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 3.7.0
>Reporter: Sebastian Marsching
>Priority: Major
>
> There is a bug in {{{}KafkaDockerWrapper{}}}, that causes {{Some( CLUSTER_ID environment variable>)}} to be used when formatting the log dir
> when Kafka is started for the first time inside a Docker container.
> More specifically, the problem is in {{{}formatStorageCmd{}}}: The code uses
> {{{}env.get("CLUSTER_ID"){}}}, but this returns an {{Option}} not a
> {{{}String{}}}.
> The code should instead check whether the environment variable is set,
> raising an exception if it is not set.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-16473) KafkaDockerWrapper uses wrong cluster ID when formatting log dir
[
https://issues.apache.org/jira/browse/KAFKA-16473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17833997#comment-17833997
]
Kuan Po Tseng commented on KAFKA-16473:
---
Thank you for the detailed description. Are you planning to address this issue?
If not, I'd be happy to take care of it.
> KafkaDockerWrapper uses wrong cluster ID when formatting log dir
>
>
> Key: KAFKA-16473
> URL: https://issues.apache.org/jira/browse/KAFKA-16473
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 3.7.0
>Reporter: Sebastian Marsching
>Priority: Major
>
> There is a bug in {{{}KafkaDockerWrapper{}}}, that causes {{Some( CLUSTER_ID environment variable>)}} to be used when formatting the log dir
> when Kafka is started for the first time inside a Docker container.
> More specifically, the problem is in {{{}formatStorageCmd{}}}: The code uses
> {{{}env.get("CLUSTER_ID"){}}}, but this returns an {{Option}} not a
> {{{}String{}}}.
> The code should instead check whether the environment variable is set,
> raising an exception if it is not set.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-16472) Integration tests in Java don't really run kraft case
[ https://issues.apache.org/jira/browse/KAFKA-16472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17833990#comment-17833990 ] Kuan Po Tseng commented on KAFKA-16472: --- And the javadoc in junit5 also mentions that https://github.com/junit-team/junit5/blob/4c0dddad1b96d4a20e92a2cd583954643ac56ac0/junit-jupiter-params/src/main/java/org/junit/jupiter/params/ParameterizedTest.java#L163 > Integration tests in Java don't really run kraft case > - > > Key: KAFKA-16472 > URL: https://issues.apache.org/jira/browse/KAFKA-16472 > Project: Kafka > Issue Type: Test >Reporter: PoAn Yang >Assignee: PoAn Yang >Priority: Major > > Following test cases don't really run kraft case. The reason is that the test > info doesn't contain parameter name, so it always returns false in > TestInfoUtils#isKRaft. > * TopicCommandIntegrationTest > * DeleteConsumerGroupsTest > * AuthorizerIntegrationTest > * DeleteOffsetsConsumerGroupCommandIntegrationTest > > We can add `options.compilerArgs += '-parameters'` after > [https://github.com/apache/kafka/blob/376e9e20dbf7c7aeb6f6f666d47932c445eb6bd1/build.gradle#L264-L273] > to fix it. > > Also, we have to add `String quorum` to cases in > DeleteOffsetsConsumerGroupCommandIntegrationTest. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16472) Integration tests in Java don't really run kraft case
[
https://issues.apache.org/jira/browse/KAFKA-16472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17833988#comment-17833988
]
Kuan Po Tseng commented on KAFKA-16472:
---
Looks like
[https://github.com/junit-team/junit5/blob/4c0dddad1b96d4a20e92a2cd583954643ac56ac0/junit-jupiter-params/src/main/java/org/junit/jupiter/params/ParameterizedTestNameFormatter.java#L93]
in this line, the scala tests can get the correct parameter name (i.e. quorum)
back while java tests could only get Optional.empty. As PoAn pointed out, add
compilerArgs {{-parameters}} can solve the missing parameter name in java tests.
> Integration tests in Java don't really run kraft case
> -
>
> Key: KAFKA-16472
> URL: https://issues.apache.org/jira/browse/KAFKA-16472
> Project: Kafka
> Issue Type: Test
>Reporter: PoAn Yang
>Assignee: PoAn Yang
>Priority: Major
>
> Following test cases don't really run kraft case. The reason is that the test
> info doesn't contain parameter name, so it always returns false in
> TestInfoUtils#isKRaft.
> * TopicCommandIntegrationTest
> * DeleteConsumerGroupsTest
> * AuthorizerIntegrationTest
> * DeleteOffsetsConsumerGroupCommandIntegrationTest
>
> We can add `options.compilerArgs += '-parameters'` after
> [https://github.com/apache/kafka/blob/376e9e20dbf7c7aeb6f6f666d47932c445eb6bd1/build.gradle#L264-L273]
> to fix it.
>
> Also, we have to add `String quorum` to cases in
> DeleteOffsetsConsumerGroupCommandIntegrationTest.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KAFKA-16455) Check partition exists before send reassignments to server in ReassignPartitionsCommand
Kuan Po Tseng created KAFKA-16455:
-
Summary: Check partition exists before send reassignments to
server in ReassignPartitionsCommand
Key: KAFKA-16455
URL: https://issues.apache.org/jira/browse/KAFKA-16455
Project: Kafka
Issue Type: Improvement
Components: tools
Reporter: Kuan Po Tseng
Assignee: Kuan Po Tseng
Currently, when executing {{kafka-reassign-partitions.sh}} with the
{{--execute}} option, if a partition number specified in the JSON file does not
exist, this check occurs only when submitting the reassignments to
{{alterPartitionReassignments}} on the server-side.
We can perform this check in advance before submitting the reassignments to the
server side.
For example, suppose we have three brokers with IDs 1001, 1002, and 1003, and a
topic named {{first_topic}} with only three partitions. And execute
{code:bash}
bin/kafka-reassign-partitions.sh
--bootstrap-server 192.168.0.128:9092
--reassignment-json-file reassignment.json
--execute
{code}
Where reassignment.json contains
{code:json}
{
"version": 1,
"partitions": [
{
"topic": "first_topic",
"partition": 20,
"replicas": [1002, 1001, 1003],
"log_dirs": ["any", "any", "any"]
}
]
}
{code}
The console outputs
{code:java}
Current partition replica assignment
{"version":1,"partitions":[]}
Save this to use as the --reassignment-json-file option during rollback
Error reassigning partition(s):
first_topic-20: The partition does not exist.
{code}
Apart from the output {{\{"version":1,"partitions":[]\}}} which doesn't provide
much help, the error {{first_topic-20: The partition does not exist.}} is
reported back to the tool from the server-side, as mentioned earlier. This
check could be moved earlier before sending reassignments to server side
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-16435) Add test for KAFKA-16428
[ https://issues.apache.org/jira/browse/KAFKA-16435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17831605#comment-17831605 ] Kuan Po Tseng commented on KAFKA-16435: --- I'm willing to take over this ! Let me add some test~ Huge thanks! > Add test for KAFKA-16428 > > > Key: KAFKA-16435 > URL: https://issues.apache.org/jira/browse/KAFKA-16435 > Project: Kafka > Issue Type: Bug >Reporter: Colin McCabe >Priority: Major > > Add a test for KAFKA-16428: Fix bug where config change notification znode > may not get created during migration #15608 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-16435) Add test for KAFKA-16428
[ https://issues.apache.org/jira/browse/KAFKA-16435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kuan Po Tseng reassigned KAFKA-16435: - Assignee: Kuan Po Tseng > Add test for KAFKA-16428 > > > Key: KAFKA-16435 > URL: https://issues.apache.org/jira/browse/KAFKA-16435 > Project: Kafka > Issue Type: Bug >Reporter: Colin McCabe >Assignee: Kuan Po Tseng >Priority: Major > > Add a test for KAFKA-16428: Fix bug where config change notification znode > may not get created during migration #15608 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-16410) kafka-leader-election / LeaderElectionCommand doesn't set exit code on error
[
https://issues.apache.org/jira/browse/KAFKA-16410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kuan Po Tseng reassigned KAFKA-16410:
-
Assignee: Kuan Po Tseng
> kafka-leader-election / LeaderElectionCommand doesn't set exit code on error
>
>
> Key: KAFKA-16410
> URL: https://issues.apache.org/jira/browse/KAFKA-16410
> Project: Kafka
> Issue Type: Bug
> Components: tools
>Affects Versions: 3.7.0
>Reporter: Greg Harris
>Assignee: Kuan Po Tseng
>Priority: Blocker
> Labels: newbie
> Fix For: 3.7.1
>
>
> The kafka-leader-election command does not set the process exit code to
> nonzero when an unexpected error occurs.
> {noformat}
> % bin/kafka-leader-election.sh --path-to-json-file /tmp/does-not-exist
> Missing required option(s): bootstrap-server, election-type
> org.apache.kafka.server.common.AdminCommandFailedException: Missing required
> option(s): bootstrap-server, election-type
> at
> org.apache.kafka.tools.LeaderElectionCommand$LeaderElectionCommandOptions.validate(LeaderElectionCommand.java:332)
> at
> org.apache.kafka.tools.LeaderElectionCommand.run(LeaderElectionCommand.java:78)
> at
> org.apache.kafka.tools.LeaderElectionCommand.main(LeaderElectionCommand.java:66)
> % echo "$?"
> 0
> {noformat}
> The exit code is sometimes set properly when other code paths cause the
> command to exit, or in versions < 3.7:
> {noformat}
> % bin/kafka-leader-election.sh
> This tool attempts to elect a new leader for a set of topic partitions. The
> type of elections supported are preferred replicas and unclean replicas.
> Option Description
> -- ---
> ...
> % echo "$?"
> 1
> {noformat}
> This appears to be a regression in 3.7.0, and since a shell script may be
> relying on the return code from this command, this is something we should fix
> in the next release.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-16410) kafka-leader-election / LeaderElectionCommand doesn't set exit code on error
[
https://issues.apache.org/jira/browse/KAFKA-16410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17830243#comment-17830243
]
Kuan Po Tseng commented on KAFKA-16410:
---
Hi [~gharris1727] , I'm willing to take over this issue, thanks for bring this
up !
> kafka-leader-election / LeaderElectionCommand doesn't set exit code on error
>
>
> Key: KAFKA-16410
> URL: https://issues.apache.org/jira/browse/KAFKA-16410
> Project: Kafka
> Issue Type: Bug
> Components: tools
>Affects Versions: 3.7.0
>Reporter: Greg Harris
>Priority: Blocker
> Labels: newbie
> Fix For: 3.7.1
>
>
> The kafka-leader-election command does not set the process exit code to
> nonzero when an unexpected error occurs.
> {noformat}
> % bin/kafka-leader-election.sh --path-to-json-file /tmp/does-not-exist
> Missing required option(s): bootstrap-server, election-type
> org.apache.kafka.server.common.AdminCommandFailedException: Missing required
> option(s): bootstrap-server, election-type
> at
> org.apache.kafka.tools.LeaderElectionCommand$LeaderElectionCommandOptions.validate(LeaderElectionCommand.java:332)
> at
> org.apache.kafka.tools.LeaderElectionCommand.run(LeaderElectionCommand.java:78)
> at
> org.apache.kafka.tools.LeaderElectionCommand.main(LeaderElectionCommand.java:66)
> % echo "$?"
> 0
> {noformat}
> The exit code is sometimes set properly when other code paths cause the
> command to exit, or in versions < 3.7:
> {noformat}
> % bin/kafka-leader-election.sh
> This tool attempts to elect a new leader for a set of topic partitions. The
> type of elections supported are preferred replicas and unclean replicas.
> Option Description
> -- ---
> ...
> % echo "$?"
> 1
> {noformat}
> This appears to be a regression in 3.7.0, and since a shell script may be
> relying on the return code from this command, this is something we should fix
> in the next release.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-16385) Segment is rolled before segment.ms or segment.bytes breached
[
https://issues.apache.org/jira/browse/KAFKA-16385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17830230#comment-17830230
]
Kuan Po Tseng commented on KAFKA-16385:
---
Gentle ping [~chia7712], [~jeqo]. I've filed another JIRA ticket KAFKA-16414 to
discuss the different behavior between `retention.ms` and `retention.bytes.
Many thanks :)
> Segment is rolled before segment.ms or segment.bytes breached
> -
>
> Key: KAFKA-16385
> URL: https://issues.apache.org/jira/browse/KAFKA-16385
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 3.5.1, 3.7.0
>Reporter: Luke Chen
>Assignee: Kuan Po Tseng
>Priority: Major
>
> Steps to reproduce:
> 0. Startup a broker with `log.retention.check.interval.ms=1000` to speed up
> the test.
> 1. Creating a topic with the config: segment.ms=7days , segment.bytes=1GB,
> retention.ms=1sec .
> 2. Send a record "aaa" to the topic
> 3. Wait for 1 second
> Will this segment will rolled? I thought no.
> But what I have tested is it will roll:
> {code:java}
> [2024-03-19 15:23:13,924] INFO [LocalLog partition=t2-1,
> dir=/tmp/kafka-logs_jbod] Rolled new log segment at offset 1 in 3 ms.
> (kafka.log.LocalLog)
> [2024-03-19 15:23:13,925] INFO [ProducerStateManager partition=t2-1] Wrote
> producer snapshot at offset 1 with 1 producer ids in 1 ms.
> (org.apache.kafka.storage.internals.log.ProducerStateManager)
> [2024-03-19 15:23:13,925] INFO [UnifiedLog partition=t2-1,
> dir=/tmp/kafka-logs_jbod] Deleting segment LogSegment(baseOffset=0, size=71,
> lastModifiedTime=1710832993131, largestRecordTimestamp=1710832992125) due to
> log retention time 1000ms breach based on the largest record timestamp in the
> segment (kafka.log.UnifiedLog)
> {code}
> The segment is rolled due to log retention time 1000ms breached, which is
> unexpected.
> Tested in v3.5.1, it has the same issue.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KAFKA-16414) Inconsistent active segment expiration behavior between retention.ms and retention.bytes
Kuan Po Tseng created KAFKA-16414: - Summary: Inconsistent active segment expiration behavior between retention.ms and retention.bytes Key: KAFKA-16414 URL: https://issues.apache.org/jira/browse/KAFKA-16414 Project: Kafka Issue Type: Improvement Affects Versions: 3.6.1 Reporter: Kuan Po Tseng Assignee: Kuan Po Tseng This is a follow up issue on KAFKA-16385. Currently, there's a difference between how retention.ms and retention.bytes handle active segment expiration: - retention.ms always expire active segment when max segment timestamp matches the condition. - retention.bytes only expire active segment when retention.bytes is configured to zero. The behavior should be either rotate active segments for both retention configurations or none at all. For more details, see https://issues.apache.org/jira/browse/KAFKA-16385?focusedCommentId=17829682=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17829682 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16385) Segment is rolled before segment.ms or segment.bytes breached
[
https://issues.apache.org/jira/browse/KAFKA-16385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17830199#comment-17830199
]
Kuan Po Tseng commented on KAFKA-16385:
---
{quote}
I, as well, agree that we should include the active segment on the retention
checks; but would like to also discuss whether we should align active segment
rotation for size-based retention as well.
{quote}
Thank you [~jeqo]. Indeed there are some inconsistent behavior between
retention.ms and retention.bytes regrading the expiration of active segments.
That might need more discussion if we want to align their behavior. Before we
conclude this discussion, we should document these differences so users don't
get confused. I've address a PR in https://github.com/apache/kafka/pull/15588
and add more description mentioned in this JIRA discussion.
> Segment is rolled before segment.ms or segment.bytes breached
> -
>
> Key: KAFKA-16385
> URL: https://issues.apache.org/jira/browse/KAFKA-16385
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 3.5.1, 3.7.0
>Reporter: Luke Chen
>Assignee: Kuan Po Tseng
>Priority: Major
>
> Steps to reproduce:
> 0. Startup a broker with `log.retention.check.interval.ms=1000` to speed up
> the test.
> 1. Creating a topic with the config: segment.ms=7days , segment.bytes=1GB,
> retention.ms=1sec .
> 2. Send a record "aaa" to the topic
> 3. Wait for 1 second
> Will this segment will rolled? I thought no.
> But what I have tested is it will roll:
> {code:java}
> [2024-03-19 15:23:13,924] INFO [LocalLog partition=t2-1,
> dir=/tmp/kafka-logs_jbod] Rolled new log segment at offset 1 in 3 ms.
> (kafka.log.LocalLog)
> [2024-03-19 15:23:13,925] INFO [ProducerStateManager partition=t2-1] Wrote
> producer snapshot at offset 1 with 1 producer ids in 1 ms.
> (org.apache.kafka.storage.internals.log.ProducerStateManager)
> [2024-03-19 15:23:13,925] INFO [UnifiedLog partition=t2-1,
> dir=/tmp/kafka-logs_jbod] Deleting segment LogSegment(baseOffset=0, size=71,
> lastModifiedTime=1710832993131, largestRecordTimestamp=1710832992125) due to
> log retention time 1000ms breach based on the largest record timestamp in the
> segment (kafka.log.UnifiedLog)
> {code}
> The segment is rolled due to log retention time 1000ms breached, which is
> unexpected.
> Tested in v3.5.1, it has the same issue.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-16385) Segment is rolled before segment.ms or segment.bytes breached
[
https://issues.apache.org/jira/browse/KAFKA-16385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17828654#comment-17828654
]
Kuan Po Tseng commented on KAFKA-16385:
---
{quote}
[~brandboat] , are you clear what you should do for this ticket?
Please let us know if you have any question.
{quote}
Thanks, I'm still poking around the source code, but sounds like we should
document the behavior mentioned in this JIRA ticket.
If I have any questions, I'll consult with you all again. Huge thanks !
> Segment is rolled before segment.ms or segment.bytes breached
> -
>
> Key: KAFKA-16385
> URL: https://issues.apache.org/jira/browse/KAFKA-16385
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 3.5.1, 3.7.0
>Reporter: Luke Chen
>Assignee: Kuan Po Tseng
>Priority: Major
>
> Steps to reproduce:
> 0. Startup a broker with `log.retention.check.interval.ms=1000` to speed up
> the test.
> 1. Creating a topic with the config: segment.ms=7days , segment.bytes=1GB,
> retention.ms=1sec .
> 2. Send a record "aaa" to the topic
> 3. Wait for 1 second
> Will this segment will rolled? I thought no.
> But what I have tested is it will roll:
> {code:java}
> [2024-03-19 15:23:13,924] INFO [LocalLog partition=t2-1,
> dir=/tmp/kafka-logs_jbod] Rolled new log segment at offset 1 in 3 ms.
> (kafka.log.LocalLog)
> [2024-03-19 15:23:13,925] INFO [ProducerStateManager partition=t2-1] Wrote
> producer snapshot at offset 1 with 1 producer ids in 1 ms.
> (org.apache.kafka.storage.internals.log.ProducerStateManager)
> [2024-03-19 15:23:13,925] INFO [UnifiedLog partition=t2-1,
> dir=/tmp/kafka-logs_jbod] Deleting segment LogSegment(baseOffset=0, size=71,
> lastModifiedTime=1710832993131, largestRecordTimestamp=1710832992125) due to
> log retention time 1000ms breach based on the largest record timestamp in the
> segment (kafka.log.UnifiedLog)
> {code}
> The segment is rolled due to log retention time 1000ms breached, which is
> unexpected.
> Tested in v3.5.1, it has the same issue.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (KAFKA-16388) add production-ready test of 3.3 - 3.6 release to MetadataVersionTest.testFromVersionString
[
https://issues.apache.org/jira/browse/KAFKA-16388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kuan Po Tseng reassigned KAFKA-16388:
-
Assignee: Kuan Po Tseng
> add production-ready test of 3.3 - 3.6 release to
> MetadataVersionTest.testFromVersionString
> ---
>
> Key: KAFKA-16388
> URL: https://issues.apache.org/jira/browse/KAFKA-16388
> Project: Kafka
> Issue Type: Test
>Reporter: Chia-Ping Tsai
>Assignee: Kuan Po Tseng
>Priority: Minor
> Labels: newbie
>
> https://github.com/apache/kafka/blob/trunk/server-common/src/test/java/org/apache/kafka/server/common/MetadataVersionTest.java#L169
> we have already released 3.3 ~ 3.6, and so they should be included by
> MetadataVersionTest.testFromVersionString
> {code:java}
> assertEquals(IBP_3_3_IV3, MetadataVersion.fromVersionString("3.3"));
> assertEquals(IBP_3_4_IV0, MetadataVersion.fromVersionString("3.4"));
> assertEquals(IBP_3_5_IV2, MetadataVersion.fromVersionString("3.5"));
> assertEquals(IBP_3_6_IV2, MetadataVersion.fromVersionString("3.6"));
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-16388) add production-ready test of 3.3 - 3.6 release to MetadataVersionTest.testFromVersionString
[
https://issues.apache.org/jira/browse/KAFKA-16388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17828542#comment-17828542
]
Kuan Po Tseng commented on KAFKA-16388:
---
I'm willing to take over this~ Thanks !
> add production-ready test of 3.3 - 3.6 release to
> MetadataVersionTest.testFromVersionString
> ---
>
> Key: KAFKA-16388
> URL: https://issues.apache.org/jira/browse/KAFKA-16388
> Project: Kafka
> Issue Type: Test
>Reporter: Chia-Ping Tsai
>Priority: Minor
> Labels: newbie
>
> https://github.com/apache/kafka/blob/trunk/server-common/src/test/java/org/apache/kafka/server/common/MetadataVersionTest.java#L169
> we have already released 3.3 ~ 3.6, and so they should be included by
> MetadataVersionTest.testFromVersionString
> {code:java}
> assertEquals(IBP_3_3_IV3, MetadataVersion.fromVersionString("3.3"));
> assertEquals(IBP_3_4_IV0, MetadataVersion.fromVersionString("3.4"));
> assertEquals(IBP_3_5_IV2, MetadataVersion.fromVersionString("3.5"));
> assertEquals(IBP_3_6_IV2, MetadataVersion.fromVersionString("3.6"));
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (KAFKA-16385) Segment is rolled before segment.ms or segment.bytes breached
[
https://issues.apache.org/jira/browse/KAFKA-16385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kuan Po Tseng reassigned KAFKA-16385:
-
Assignee: Kuan Po Tseng
> Segment is rolled before segment.ms or segment.bytes breached
> -
>
> Key: KAFKA-16385
> URL: https://issues.apache.org/jira/browse/KAFKA-16385
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 3.7.0
>Reporter: Luke Chen
>Assignee: Kuan Po Tseng
>Priority: Major
>
> Steps to reproduce:
> 1. Creating a topic with the config: segment.ms=7days , retention.ms=1sec .
> 2. Send a record "aaa" to the topic
> 3. Wait for 1 second
> Will this segment will rolled? I thought no.
> But what I have tested is it will roll:
> {code:java}
> [2024-03-19 15:23:13,924] INFO [LocalLog partition=t2-1,
> dir=/tmp/kafka-logs_jbod] Rolled new log segment at offset 1 in 3 ms.
> (kafka.log.LocalLog)
> [2024-03-19 15:23:13,925] INFO [ProducerStateManager partition=t2-1] Wrote
> producer snapshot at offset 1 with 1 producer ids in 1 ms.
> (org.apache.kafka.storage.internals.log.ProducerStateManager)
> [2024-03-19 15:23:13,925] INFO [UnifiedLog partition=t2-1,
> dir=/tmp/kafka-logs_jbod] Deleting segment LogSegment(baseOffset=0, size=71,
> lastModifiedTime=1710832993131, largestRecordTimestamp=1710832992125) due to
> log retention time 1000ms breach based on the largest record timestamp in the
> segment (kafka.log.UnifiedLog)
> {code}
> The segment is rolled due to log retention time 1000ms breached, which is
> unexpected.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-16385) Segment is rolled before segment.ms or segment.bytes breached
[
https://issues.apache.org/jira/browse/KAFKA-16385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17828227#comment-17828227
]
Kuan Po Tseng commented on KAFKA-16385:
---
I'm willing to take over this ! Many thanks !
> Segment is rolled before segment.ms or segment.bytes breached
> -
>
> Key: KAFKA-16385
> URL: https://issues.apache.org/jira/browse/KAFKA-16385
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 3.7.0
>Reporter: Luke Chen
>Priority: Major
>
> Steps to reproduce:
> 1. Creating a topic with the config: segment.ms=7days , retention.ms=1sec .
> 2. Send a record "aaa" to the topic
> 3. Wait for 1 second
> Will this segment will rolled? I thought no.
> But what I have tested is it will roll:
> {code:java}
> [2024-03-19 15:23:13,924] INFO [LocalLog partition=t2-1,
> dir=/tmp/kafka-logs_jbod] Rolled new log segment at offset 1 in 3 ms.
> (kafka.log.LocalLog)
> [2024-03-19 15:23:13,925] INFO [ProducerStateManager partition=t2-1] Wrote
> producer snapshot at offset 1 with 1 producer ids in 1 ms.
> (org.apache.kafka.storage.internals.log.ProducerStateManager)
> [2024-03-19 15:23:13,925] INFO [UnifiedLog partition=t2-1,
> dir=/tmp/kafka-logs_jbod] Deleting segment LogSegment(baseOffset=0, size=71,
> lastModifiedTime=1710832993131, largestRecordTimestamp=1710832992125) due to
> log retention time 1000ms breach based on the largest record timestamp in the
> segment (kafka.log.UnifiedLog)
> {code}
> The segment is rolled due to log retention time 1000ms breached, which is
> unexpected.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-16376) Fix flaky ReplicaManagerTest.testRemoteFetchExpiresPerSecMetric
[
https://issues.apache.org/jira/browse/KAFKA-16376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17827824#comment-17827824
]
Kuan Po Tseng commented on KAFKA-16376:
---
Correct me if I'm wrong, but this appears to be a duplicate of KAFKA-16323.
> Fix flaky ReplicaManagerTest.testRemoteFetchExpiresPerSecMetric
> ---
>
> Key: KAFKA-16376
> URL: https://issues.apache.org/jira/browse/KAFKA-16376
> Project: Kafka
> Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: PoAn Yang
>Priority: Major
>
> {quote}
> [2024-03-13T17:22:47.835Z] > Task :core:test
> [2024-03-13T17:22:47.835Z]
> kafka.server.ReplicaManagerTest.testRemoteFetchExpiresPerSecMetric() failed,
> log available in
> /home/jenkins/jenkins-agent/workspace/Kafka_kafka-pr_PR-15474/core/build/reports/testOutput/kafka.server.ReplicaManagerTest.testRemoteFetchExpiresPerSecMetric().test.stdout
> [2024-03-13T17:22:47.835Z]
> [2024-03-13T17:22:49.409Z] Gradle Test Run :core:test > Gradle Test Executor
> 97 > ReplicaManagerTest > testRemoteFetchExpiresPerSecMetric() FAILED
> [2024-03-13T17:22:49.409Z] org.opentest4j.AssertionFailedError: The
> ExpiresPerSec value is not incremented. Current value is: 0
> [2024-03-13T17:22:49.409Z] at
> org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:38)
> [2024-03-13T17:22:49.409Z] at
> org.junit.jupiter.api.Assertions.fail(Assertions.java:138)
> [2024-03-13T17:22:49.409Z] at
> kafka.server.ReplicaManagerTest.testRemoteFetchExpiresPerSecMetric(ReplicaManagerTest.scala:4174)
> {quote}
> https://ci-builds.apache.org/blue/rest/organizations/jenkins/pipelines/Kafka/pipelines/kafka-pr/branches/PR-15474/runs/12/nodes/9/steps/88/log/?start=0
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-16263) Add Kafka Streams docs about available listeners/callback
[ https://issues.apache.org/jira/browse/KAFKA-16263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17827779#comment-17827779 ] Kuan Po Tseng commented on KAFKA-16263: --- May I take over this issue ? I'm quite interested in Kafka Streams :) > Add Kafka Streams docs about available listeners/callback > - > > Key: KAFKA-16263 > URL: https://issues.apache.org/jira/browse/KAFKA-16263 > Project: Kafka > Issue Type: Task > Components: docs, streams >Reporter: Matthias J. Sax >Priority: Minor > Labels: beginner, newbie > > Kafka Streams allows to register all kind of listeners and callback (eg, > uncaught-exception-handler, restore-listeners, etc) but those are not in the > documentation. > A good place might be > [https://kafka.apache.org/documentation/streams/developer-guide/running-app.html] > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16377) Fix flaky HighAvailabilityTaskAssignorIntegrationTest.shouldScaleOutWithWarmupTasksAndPersistentStores
[
https://issues.apache.org/jira/browse/KAFKA-16377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17827766#comment-17827766
]
Kuan Po Tseng commented on KAFKA-16377:
---
I'm willing to take over this issue, many thanks !
> Fix flaky
> HighAvailabilityTaskAssignorIntegrationTest.shouldScaleOutWithWarmupTasksAndPersistentStores
> --
>
> Key: KAFKA-16377
> URL: https://issues.apache.org/jira/browse/KAFKA-16377
> Project: Kafka
> Issue Type: Bug
> Components: streams, unit tests
>Reporter: Chia-Ping Tsai
>Priority: Major
>
> {quote}
> [2024-03-13T16:07:11.125Z] Gradle Test Run :streams:test > Gradle Test
> Executor 95 > HighAvailabilityTaskAssignorIntegrationTest >
> shouldScaleOutWithWarmupTasksAndPersistentStores(String, TestInfo) >
> "shouldScaleOutWithWarmupTasksAndPersistentStores(String,
> TestInfo).balance_subtopology" FAILED
> [2024-03-13T16:07:11.125Z] java.lang.AssertionError: the first assignment
> after adding a node should be unstable while we warm up the state.
> [2024-03-13T16:07:11.125Z] at
> org.apache.kafka.streams.integration.HighAvailabilityTaskAssignorIntegrationTest.assertFalseNoRetry(HighAvailabilityTaskAssignorIntegrationTest.java:310)
> [2024-03-13T16:07:11.125Z] at
> org.apache.kafka.streams.integration.HighAvailabilityTaskAssignorIntegrationTest.lambda$shouldScaleOutWithWarmupTasks$7(HighAvailabilityTaskAssignorIntegrationTest.java:237)
> [2024-03-13T16:07:11.125Z] at
> org.apache.kafka.test.TestUtils.lambda$waitForCondition$3(TestUtils.java:395)
> [2024-03-13T16:07:11.125Z] at
> org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:443)
> [2024-03-13T16:07:11.125Z] at
> org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:392)
> [2024-03-13T16:07:11.125Z] at
> org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:376)
> [2024-03-13T16:07:11.125Z] at
> org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:366)
> [2024-03-13T16:07:11.125Z] at
> org.apache.kafka.streams.integration.HighAvailabilityTaskAssignorIntegrationTest.shouldScaleOutWithWarmupTasks(HighAvailabilityTaskAssignorIntegrationTest.java:232)
> [2024-03-13T16:07:11.125Z] at
> org.apache.kafka.streams.integration.HighAvailabilityTaskAssignorIntegrationTest.shouldScaleOutWithWarmupTasksAndPersistentStores(HighAvailabilityTaskAssignorIntegrationTest.java:130)
> {quote}
> https://ci-builds.apache.org/blue/rest/organizations/jenkins/pipelines/Kafka/pipelines/kafka-pr/branches/PR-15474/runs/12/nodes/9/steps/88/log/?start=0
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (KAFKA-16377) Fix flaky HighAvailabilityTaskAssignorIntegrationTest.shouldScaleOutWithWarmupTasksAndPersistentStores
[
https://issues.apache.org/jira/browse/KAFKA-16377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kuan Po Tseng reassigned KAFKA-16377:
-
Assignee: Kuan Po Tseng
> Fix flaky
> HighAvailabilityTaskAssignorIntegrationTest.shouldScaleOutWithWarmupTasksAndPersistentStores
> --
>
> Key: KAFKA-16377
> URL: https://issues.apache.org/jira/browse/KAFKA-16377
> Project: Kafka
> Issue Type: Bug
> Components: streams, unit tests
>Reporter: Chia-Ping Tsai
>Assignee: Kuan Po Tseng
>Priority: Major
>
> {quote}
> [2024-03-13T16:07:11.125Z] Gradle Test Run :streams:test > Gradle Test
> Executor 95 > HighAvailabilityTaskAssignorIntegrationTest >
> shouldScaleOutWithWarmupTasksAndPersistentStores(String, TestInfo) >
> "shouldScaleOutWithWarmupTasksAndPersistentStores(String,
> TestInfo).balance_subtopology" FAILED
> [2024-03-13T16:07:11.125Z] java.lang.AssertionError: the first assignment
> after adding a node should be unstable while we warm up the state.
> [2024-03-13T16:07:11.125Z] at
> org.apache.kafka.streams.integration.HighAvailabilityTaskAssignorIntegrationTest.assertFalseNoRetry(HighAvailabilityTaskAssignorIntegrationTest.java:310)
> [2024-03-13T16:07:11.125Z] at
> org.apache.kafka.streams.integration.HighAvailabilityTaskAssignorIntegrationTest.lambda$shouldScaleOutWithWarmupTasks$7(HighAvailabilityTaskAssignorIntegrationTest.java:237)
> [2024-03-13T16:07:11.125Z] at
> org.apache.kafka.test.TestUtils.lambda$waitForCondition$3(TestUtils.java:395)
> [2024-03-13T16:07:11.125Z] at
> org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:443)
> [2024-03-13T16:07:11.125Z] at
> org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:392)
> [2024-03-13T16:07:11.125Z] at
> org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:376)
> [2024-03-13T16:07:11.125Z] at
> org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:366)
> [2024-03-13T16:07:11.125Z] at
> org.apache.kafka.streams.integration.HighAvailabilityTaskAssignorIntegrationTest.shouldScaleOutWithWarmupTasks(HighAvailabilityTaskAssignorIntegrationTest.java:232)
> [2024-03-13T16:07:11.125Z] at
> org.apache.kafka.streams.integration.HighAvailabilityTaskAssignorIntegrationTest.shouldScaleOutWithWarmupTasksAndPersistentStores(HighAvailabilityTaskAssignorIntegrationTest.java:130)
> {quote}
> https://ci-builds.apache.org/blue/rest/organizations/jenkins/pipelines/Kafka/pipelines/kafka-pr/branches/PR-15474/runs/12/nodes/9/steps/88/log/?start=0
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (KAFKA-16283) RoundRobinPartitioner will only send to half of the partitions in a topic
[
https://issues.apache.org/jira/browse/KAFKA-16283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kuan-Po Tseng reassigned KAFKA-16283:
-
Assignee: Kuan-Po Tseng
> RoundRobinPartitioner will only send to half of the partitions in a topic
> -
>
> Key: KAFKA-16283
> URL: https://issues.apache.org/jira/browse/KAFKA-16283
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 3.1.0, 3.0.0, 3.6.1
>Reporter: Luke Chen
>Assignee: Kuan-Po Tseng
>Priority: Major
>
> When using `org.apache.kafka.clients.producer.RoundRobinPartitioner`, we
> expect data are sent to all partitions in round-robin manner. But we found
> there are only half of the partitions got the data. This causes half of the
> resources(storage, consumer...) are wasted.
> {code:java}
> > bin/kafka-topics.sh --create --topic quickstart-events4 --bootstrap-server
> > localhost:9092 --partitions 2
> Created topic quickstart-events4.
> # send 1000 records to the topic, expecting 500 records in partition0, and
> 500 records in partition1
> > bin/kafka-producer-perf-test.sh --topic quickstart-events4 --num-records
> > 1000 --record-size 1024 --throughput -1 --producer-props
> > bootstrap.servers=localhost:9092
> > partitioner.class=org.apache.kafka.clients.producer.RoundRobinPartitioner
> 1000 records sent, 6535.947712 records/sec (6.38 MB/sec), 2.88 ms avg
> latency, 121.00 ms max latency, 2 ms 50th, 7 ms 95th, 10 ms 99th, 121 ms
> 99.9th.
> > ls -al /tmp/kafka-logs/quickstart-events4-1
> total 24
> drwxr-xr-x 7 lukchen wheel 224 2 20 19:53 .
> drwxr-xr-x 70 lukchen wheel 2240 2 20 19:53 ..
> -rw-r--r-- 1 lukchen wheel 10485760 2 20 19:53 .index
> -rw-r--r-- 1 lukchen wheel 1037819 2 20 19:53 .log
> -rw-r--r-- 1 lukchen wheel 10485756 2 20 19:53
> .timeindex
> -rw-r--r-- 1 lukchen wheel 8 2 20 19:53 leader-epoch-checkpoint
> -rw-r--r-- 1 lukchen wheel43 2 20 19:53 partition.metadata
> # No records in partition 1
> > ls -al /tmp/kafka-logs/quickstart-events4-0
> total 8
> drwxr-xr-x 7 lukchen wheel 224 2 20 19:53 .
> drwxr-xr-x 70 lukchen wheel 2240 2 20 19:53 ..
> -rw-r--r-- 1 lukchen wheel 10485760 2 20 19:53 .index
> -rw-r--r-- 1 lukchen wheel 0 2 20 19:53 .log
> -rw-r--r-- 1 lukchen wheel 10485756 2 20 19:53
> .timeindex
> -rw-r--r-- 1 lukchen wheel 0 2 20 19:53 leader-epoch-checkpoint
> -rw-r--r-- 1 lukchen wheel43 2 20 19:53 partition.metadata
> {code}
> Tested in kafka 3.0.0, 3.2.3, and the latest trunk, they all have the same
> issue. It should already exist for a long time.
>
> Had a quick look, it's because we will abortOnNewBatch each time when new
> batch created.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (KAFKA-12187) replace assertTrue(obj instanceof X) by assertInstanceOf when we update to JUnit 5.8
[
https://issues.apache.org/jira/browse/KAFKA-12187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kuan-Po Tseng reassigned KAFKA-12187:
-
Assignee: Kuan-Po Tseng (was: Chia-Ping Tsai)
> replace assertTrue(obj instanceof X) by assertInstanceOf when we update to
> JUnit 5.8
>
>
> Key: KAFKA-12187
> URL: https://issues.apache.org/jira/browse/KAFKA-12187
> Project: Kafka
> Issue Type: Sub-task
>Reporter: Chia-Ping Tsai
>Assignee: Kuan-Po Tseng
>Priority: Minor
>
> see [https://github.com/apache/kafka/pull/9874#discussion_r556547909]
>
> {quote}Yeah, for existing code improvements (versus code introduced by this
> change), let's do it via a different PR. For this particular issue, we can
> probably wait for JUnit 5.8 and use:
> {quote}
> * New assertInstanceOf methods as a replacement for assertTrue(obj instanceof
> X) which provide better error messages comparable to those of assertThrows.
> related PR: https://github.com/junit-team/junit5/pull/2499
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-16232) kafka hangs forever in the starting process if the authorizer future is not returned
[ https://issues.apache.org/jira/browse/KAFKA-16232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17822739#comment-17822739 ] Kuan-Po Tseng commented on KAFKA-16232: --- Thank you [~showuon] , but I can't assign this issue to myself, could you help me ? Thanks again ! > kafka hangs forever in the starting process if the authorizer future is not > returned > > > Key: KAFKA-16232 > URL: https://issues.apache.org/jira/browse/KAFKA-16232 > Project: Kafka > Issue Type: Improvement >Affects Versions: 3.6.1 >Reporter: Luke Chen >Priority: Major > > For security reason, during broker startup, we will wait until all ACL > entries loaded before starting serving requests. But recently, we > accidentally set standardAuthorizer to ZK broker, and then, the broker never > enters RUNNING state because it's waiting for the standardAuthorizer future > completion. Of course this is a human error to set the wrong configuration, > but it'd be better we could handle this case better. Suggestions: > 1. set timeout for authorizer future waiting (how long is long enough?) > 2. add logs before and after future waiting, to allow admin to know we're > waiting for the authorizer future. > We can start with (2), and thinking about (1) later. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16232) kafka hangs forever in the starting process if the authorizer future is not returned
[ https://issues.apache.org/jira/browse/KAFKA-16232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17822532#comment-17822532 ] Kuan-Po Tseng commented on KAFKA-16232: --- gentle ping [~showuon] ~ I'm willing to take over this issue, as you mentioned, I'll try to start with (2) first. Many thanks :) > kafka hangs forever in the starting process if the authorizer future is not > returned > > > Key: KAFKA-16232 > URL: https://issues.apache.org/jira/browse/KAFKA-16232 > Project: Kafka > Issue Type: Improvement >Affects Versions: 3.6.1 >Reporter: Luke Chen >Priority: Major > > For security reason, during broker startup, we will wait until all ACL > entries loaded before starting serving requests. But recently, we > accidentally set standardAuthorizer to ZK broker, and then, the broker never > enters RUNNING state because it's waiting for the standardAuthorizer future > completion. Of course this is a human error to set the wrong configuration, > but it'd be better we could handle this case better. Suggestions: > 1. set timeout for authorizer future waiting (how long is long enough?) > 2. add logs before and after future waiting, to allow admin to know we're > waiting for the authorizer future. > We can start with (2), and thinking about (1) later. -- This message was sent by Atlassian Jira (v8.20.10#820010)
