[
https://issues.apache.org/jira/browse/KAFKA-15682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kamal Chandraprakash updated KAFKA-15682:
-----------------------------------------
Description:
One of the implementation of RemoteLogMetadataManager is
TopicBasedRemoteLogMetadataManager which uses an internal Kafka topic
{{__remote_log_metadata}} to store the metadata about the remote log segments.
Unlike other internal topics which are compaction enabled, this topic is not
enabled with compaction and retention is set to unlimited.
Keeping this internal topic retention to unlimited is not practical in real
world use-case where the topic local disk usage footprint grow huge over a
period of time.
It is assumed that the user will set the retention to a reasonable time such
that it is the max of all the user-created topics (max + X). We can't just rely
on it and need an assertion before deleting the internal
{{__remote_log_metadata}} segments, otherwise there will be dangling remote log
segments which won't be cleared once all the brokers are restarted post the
topic truncation.
was:
One of the implementation of RemoteLogMetadataManager is
TopicBasedRemoteLogMetadataManager which uses an internal Kafka topic
{{__remote_log_metadata}}. Unlike other internal topics which are compaction
enabled, this topic is not enabled with compaction and retention is set to
unlimited.
Keeping this internal topic retention to unlimited is not practical in real
world use-case where the topic local disk usage footprint grow huge over a
period of time.
It is assumed that the user will set the retention to a reasonable time such
that it is the max of all the user-created topics (max + X). We can't just rely
on it and need an assertion before deleting the internal
{{__remote_log_metadata}} segments, otherwise there will be dangling remote log
segments which won't be cleared once all the brokers are restarted post the
topic truncation.
> Ensure internal remote log metadata topic does not expire its segments before
> deleting user-topic segments
> ----------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-15682
> URL: https://issues.apache.org/jira/browse/KAFKA-15682
> Project: Kafka
> Issue Type: Task
> Reporter: Kamal Chandraprakash
> Priority: Major
>
> One of the implementation of RemoteLogMetadataManager is
> TopicBasedRemoteLogMetadataManager which uses an internal Kafka topic
> {{__remote_log_metadata}} to store the metadata about the remote log
> segments. Unlike other internal topics which are compaction enabled, this
> topic is not enabled with compaction and retention is set to unlimited.
> Keeping this internal topic retention to unlimited is not practical in real
> world use-case where the topic local disk usage footprint grow huge over a
> period of time.
> It is assumed that the user will set the retention to a reasonable time such
> that it is the max of all the user-created topics (max + X). We can't just
> rely on it and need an assertion before deleting the internal
> {{__remote_log_metadata}} segments, otherwise there will be dangling remote
> log segments which won't be cleared once all the brokers are restarted post
> the topic truncation.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
