[
https://issues.apache.org/jira/browse/KAFKA-15086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17909065#comment-17909065
]
Zhijian Chen edited comment on KAFKA-15086 at 12/31/24 12:59 PM:
-----------------------------------------------------------------
I encountered the same problem, it would take a long time to start the worker
or checkpoint task. I also read the relevant discussion of
KIP-943(https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=255073470).
In fact,
KIP-605(https://cwiki.apache.org/confluence/display/KAFKA/KIP-605%3A+Expand+Connect+Worker+Internal+Topic+Settings)
already supports the configuration of internal topic. mm2 now has the ability
to set the segment bytes of the offset topic.
In my opinion, the key point of the problem is that there is no way to
configure segment bytes for the internal topic used by mm2, such as offset
sync. We can completely refer to KIP-605 to do the same implementation.
As for default values, I think we can give a configuration value and put it in
the sample of the mm2 configuration file instead of writing it in code.
For stock users, I think it is still feasible for users to manually set the
segment bytes of the topic, instead of forcing us to change the user's
configuration when the code is started, and it is best for users to control it.
In fact, I've already done the implementation, hoping to merge back into the
main branch of kafka.
My demo logs:
1、start worker took 3263 ms.
[2024-12-31 20:32:40,364] INFO Worker starting
(org.apache.kafka.connect.runtime.Worker:233)
[2024-12-31 20:32:43,885] INFO read to end,
assignment=[mm2-offsets.A.internal-3, mm2-offsets.A.internal-1,
mm2-offsets.A.internal-24, mm2-offsets.A.internal-22,
mm2-offsets.A.internal-15, mm2-offsets.A.internal-13,
mm2-offsets.A.internal-19, mm2-offsets.A.internal-17, mm2-offsets.A.internal-7,
mm2-offsets.A.internal-5, mm2-offsets.A.internal-11, mm2-offsets.A.internal-9,
mm2-offsets.A.internal-0, mm2-offsets.A.internal-4, mm2-offsets.A.internal-2,
mm2-offsets.A.internal-23, mm2-offsets.A.internal-21,
mm2-offsets.A.internal-16, mm2-offsets.A.internal-14,
mm2-offsets.A.internal-20, mm2-offsets.A.internal-18, mm2-offsets.A.internal-8,
mm2-offsets.A.internal-6, mm2-offsets.A.internal-12,
mm2-offsets.A.internal-10], count=70184, usedMs=3263
(org.apache.kafka.connect.util.KafkaBasedLog:525)
[2024-12-31 20:32:43,885] INFO Finished reading offsets topic and starting
KafkaOffsetBackingStore
(org.apache.kafka.connect.storage.KafkaOffsetBackingStore:249)
[2024-12-31 20:32:43,888] INFO Worker started
(org.apache.kafka.connect.runtime.Worker:243)
2、start checkpoint task took 4386 ms.
[2024-12-31 20:41:02,262] INFO [MirrorCheckpointConnector|task-0] read to end,
assignment=[mm2-offset-syncs.A.internal-0], count=311839, usedMs=4160
(org.apache.kafka.connect.util.KafkaBasedLog:525)
[2024-12-31 20:41:02,262] INFO [MirrorCheckpointConnector|task-0] Finished
reading KafkaBasedLog for topic mm2-offset-syncs.A.internal
(org.apache.kafka.connect.util.KafkaBasedLog:311)
[2024-12-31 20:41:02,262] INFO [MirrorCheckpointConnector|task-0] Started
KafkaBasedLog for topic mm2-offset-syncs.A.internal
(org.apache.kafka.connect.util.KafkaBasedLog:313)
[2024-12-31 20:41:02,263] INFO [MirrorCheckpointConnector|task-0] starting
checkpoint and offset sync stores took 4386 ms
(org.apache.kafka.connect.mirror.Scheduler:99)
was (Author: JIRAUSER307474):
I encountered the same problem, it would take a long time to start the worker
or checkpoint task. I also read the relevant discussion of KIP-943. In fact,
KIP-605 already supports the configuration of internal topic. mm2 now has the
ability to set the segment bytes of the offset topic.
In my opinion, the key point of the problem is that there is no way to
configure segment bytes for the internal topic used by mm2, such as offset
sync. We can completely refer to KIP-605 to do the same implementation.
As for default values, I think we can give a configuration value and put it in
the sample of the mm2 configuration file instead of writing it in code.
For stock users, I think it is still feasible for users to manually set the
segment bytes of the topic, instead of forcing us to change the user's
configuration when the code is started, and it is best for users to control it.
In fact, I've already done the implementation, hoping to merge back into the
main branch of kafka.
My demo logs:
1、start worker took 3263 ms.
[2024-12-31 20:32:40,364] INFO Worker starting
(org.apache.kafka.connect.runtime.Worker:233)
[2024-12-31 20:32:43,885] INFO read to end,
assignment=[mm2-offsets.A.internal-3, mm2-offsets.A.internal-1,
mm2-offsets.A.internal-24, mm2-offsets.A.internal-22,
mm2-offsets.A.internal-15, mm2-offsets.A.internal-13,
mm2-offsets.A.internal-19, mm2-offsets.A.internal-17, mm2-offsets.A.internal-7,
mm2-offsets.A.internal-5, mm2-offsets.A.internal-11, mm2-offsets.A.internal-9,
mm2-offsets.A.internal-0, mm2-offsets.A.internal-4, mm2-offsets.A.internal-2,
mm2-offsets.A.internal-23, mm2-offsets.A.internal-21,
mm2-offsets.A.internal-16, mm2-offsets.A.internal-14,
mm2-offsets.A.internal-20, mm2-offsets.A.internal-18, mm2-offsets.A.internal-8,
mm2-offsets.A.internal-6, mm2-offsets.A.internal-12,
mm2-offsets.A.internal-10], count=70184, usedMs=3263
(org.apache.kafka.connect.util.KafkaBasedLog:525)
[2024-12-31 20:32:43,885] INFO Finished reading offsets topic and starting
KafkaOffsetBackingStore
(org.apache.kafka.connect.storage.KafkaOffsetBackingStore:249)
[2024-12-31 20:32:43,888] INFO Worker started
(org.apache.kafka.connect.runtime.Worker:243)
2、start checkpoint task took 4386 ms.
[2024-12-31 20:41:02,262] INFO [MirrorCheckpointConnector|task-0] read to end,
assignment=[mm2-offset-syncs.A.internal-0], count=311839, usedMs=4160
(org.apache.kafka.connect.util.KafkaBasedLog:525)
[2024-12-31 20:41:02,262] INFO [MirrorCheckpointConnector|task-0] Finished
reading KafkaBasedLog for topic mm2-offset-syncs.A.internal
(org.apache.kafka.connect.util.KafkaBasedLog:311)
[2024-12-31 20:41:02,262] INFO [MirrorCheckpointConnector|task-0] Started
KafkaBasedLog for topic mm2-offset-syncs.A.internal
(org.apache.kafka.connect.util.KafkaBasedLog:313)
[2024-12-31 20:41:02,263] INFO [MirrorCheckpointConnector|task-0] starting
checkpoint and offset sync stores took 4386 ms
(org.apache.kafka.connect.mirror.Scheduler:99)
> The unreasonable segment size setting of the internal topics in MM2 may cause
> the worker startup time to be too long
> --------------------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-15086
> URL: https://issues.apache.org/jira/browse/KAFKA-15086
> Project: Kafka
> Issue Type: Improvement
> Components: mirrormaker
> Affects Versions: 3.4.1
> Reporter: hudeqi
> Assignee: hudeqi
> Priority: Major
> Labels: kip-943
> Attachments: WechatIMG364.jpeg, WechatIMG365.jpeg, WechatIMG366.jpeg
>
>
> As the config 'segment.bytes' for topics related MM2(such as
> offset.storage.topic, config.storage.topic,status.storage.topic), if
> following the default configuration of the broker or set it larger, then when
> the MM cluster runs many and complicated tasks, especially the log volume of
> the topic 'offset.storage.topic' is very large, it will affect the restart
> speed of the MM workers.
> After investigation, the reason is that a consumer needs to be started to
> read the data of ‘offset.storage.topic’ at startup. Although this topic is
> set to compact, if the 'segment size' is set to a large value, such as the
> default value of 1G, then this topic may have tens of gigabytes of data that
> cannot be compacted and has to be read from the earliest (because the active
> segment cannot be cleaned), which will consume a lot of time (in our online
> environment, we found that this topic stores 13G of data, it took nearly half
> an hour for all the data to be consumed), which caused the worker to be
> unable to start and execute tasks for a long time.
> Of course, the number of consumer threads can also be adjusted, but I think
> it may be easier to reduce the 'segment size', for example, refer to the
> default value of __consumer_offsets: 100MB
>
> The first picture in the attachment is the log size stored in the internal
> topic, the second one is the time when ‘offset.storage.topic’ starts to be
> read, and the third one is the time when ‘offset.storage.topic’ being read
> finished. It took about 23 minutes in total.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
