[ https://issues.apache.org/jira/browse/KAFKA-13996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Luke Chen updated KAFKA-13996: ------------------------------ Fix Version/s: 3.3.0 > log.cleaner.io.max.bytes.per.second cannot be changed dynamically > ------------------------------------------------------------------ > > Key: KAFKA-13996 > URL: https://issues.apache.org/jira/browse/KAFKA-13996 > Project: Kafka > Issue Type: Bug > Components: config, core, log cleaner > Affects Versions: 3.2.0 > Reporter: Tomonari Yamashita > Assignee: Tomonari Yamashita > Priority: Major > Fix For: 3.3.0 > > > - log.cleaner.io.max.bytes.per.second cannot be changed dynamically using > bin/kafka-configs.sh > - Reproduction procedure: > -# Create a topic with cleanup.policy=compact > {code:java} > bin/kafka-topics.sh --bootstrap-server localhost:9092 --create > --replication-factor 1 --partitions 1 --topic my-topic --config > cleanup.policy=compact --config cleanup.policy=compact --config > segment.bytes=104857600 --config compression.type=producer > {code} > -# Change log.cleaner.io.max.bytes.per.second=10485760 using > bin/kafka-configs.sh > {code:java} > bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type brokers > --entity-default --alter --add-config > log.cleaner.io.max.bytes.per.second=10485760 > {code} > -# Send enough messages(> segment.bytes=104857600) to activate Log Cleaner > -# logs/log-cleaner.log, configuration by > log.cleaner.io.max.bytes.per.second=10485760 is not reflected and Log Cleaner > does not slow down (>= log.cleaner.io.max.bytes.per.second=10485760). > {code:java} > [2022-06-15 14:52:14,988] INFO [kafka-log-cleaner-thread-0]: > Log cleaner thread 0 cleaned log my-topic-0 (dirty section = [39786, > 81666]) > 3,999.0 MB of log processed in 2.7 seconds (1,494.4 MB/sec). > Indexed 3,998.9 MB in 0.9 seconds (4,218.2 Mb/sec, 35.4% of total > time) > Buffer utilization: 0.0% > Cleaned 3,999.0 MB in 1.7 seconds (2,314.2 Mb/sec, 64.6% of total > time) > Start size: 3,999.0 MB (41,881 messages) > End size: 0.1 MB (1 messages) > 100.0% size reduction (100.0% fewer messages) > (kafka.log.LogCleaner) > {code} > - Problem cause: > -- log.cleaner.io.max.bytes.per.second is used in Throttler in LogCleaner, > however, it is only passed to Throttler at initialization time. > --- > https://github.com/apache/kafka/blob/4380eae7ceb840dd93fee8ec90cd89a72bad7a3f/core/src/main/scala/kafka/log/LogCleaner.scala#L107-L112 > -- Need to change Throttler configuration value at reconfigure() of > LogCleaner. > --- > https://github.com/apache/kafka/blob/4380eae7ceb840dd93fee8ec90cd89a72bad7a3f/core/src/main/scala/kafka/log/LogCleaner.scala#L192-L196 > - A workaround is that restarting every broker adding > log.cleaner.io.max.bytes.per.second to config/server.properties -- This message was sent by Atlassian Jira (v8.20.10#820010)