[ https://issues.apache.org/jira/browse/KAFKA-15682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kamal Chandraprakash reassigned KAFKA-15682: -------------------------------------------- Assignee: Kamal Chandraprakash > Ensure internal remote log metadata topic does not expire its segments before > deleting user-topic segments > ---------------------------------------------------------------------------------------------------------- > > Key: KAFKA-15682 > URL: https://issues.apache.org/jira/browse/KAFKA-15682 > Project: Kafka > Issue Type: Task > Reporter: Kamal Chandraprakash > Assignee: Kamal Chandraprakash > Priority: Major > > One of the implementation of RemoteLogMetadataManager is > TopicBasedRemoteLogMetadataManager which uses an internal Kafka topic > {{__remote_log_metadata}} to store the metadata about the remote log > segments. Unlike other internal topics which are compaction enabled, this > topic is not enabled with compaction and retention is set to unlimited. > Keeping this internal topic retention to unlimited is not practical in real > world use-case where the topic local disk usage footprint grow huge over a > period of time. > It is assumed that the user will set the retention to a reasonable time such > that it is the max of all the user-created topics (max + X). We can't just > rely on the assumption and need an assertion to ensure that the internal > {{__remote_log_metadata}} segments are not eligible for deletion before the > expiry of all the relevant user-topic uploaded remote-log-segments , > otherwise there will be dangling remote-log-segments which won't be cleared > once all the brokers are restarted post the internal topic retention cleanup. > See the discussion thread: > https://github.com/apache/kafka/pull/14576#discussion_r1368576126 -- This message was sent by Atlassian Jira (v8.20.10#820010)