[
https://issues.apache.org/jira/browse/KAFKA-14038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17751076#comment-17751076
]
Christo Lolov edited comment on KAFKA-14038 at 8/4/23 10:37 AM:
----------------------------------------------------------------
The remaining part of this ticket is blocked on merging
https://github.com/apache/kafka/pull/13561
was (Author: christo_lolov):
The remaining part of this ticket is blocked on merging
https://issues.apache.org/jira/browse/KAFKA-14038
> Optimize calculation of size for log in remote tier
> ---------------------------------------------------
>
> Key: KAFKA-14038
> URL: https://issues.apache.org/jira/browse/KAFKA-14038
> Project: Kafka
> Issue Type: Improvement
> Components: core
> Reporter: Divij Vaidya
> Assignee: Christo Lolov
> Priority: Major
> Labels: KIP-405, needs-kip
> Fix For: 3.6.0
>
>
> {color:#24292f}As per the Tiered Storage feature introduced in
> [KIP-405|https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage],
> users can configure the retention of remote tier based on time, by size, or
> both. The work of computing the log segments to be deleted based on the
> retention config is [owned by
> RemoteLogManager|https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage#KIP405:KafkaTieredStorage-1.RemoteLogManager(RLM)ThreadPool]
> (RLM).{color}
> {color:#24292f}To compute remote segments eligible for deletion based on
> retention by size config, {color}RLM needs to compute the
> {{total_remote_log_size}} i.e. the total size of logs available in the remote
> tier for that topic-partition. RLM could use the
> {{RemoteLogMetadataManager.listRemoteLogSegments()}} to fetch metadata for
> all the remote segments and then aggregate the segment sizes by using
> {{{}RemoteLogSegmentMetadata.segmentSizeInBytes(){}}}to find the total log
> size stored in the remote tier.
> The above method involves iterating through all metadata of all the segments
> i.e. O({color:#24292f}num_remote_segments{color}) on each execution of RLM
> thread. {color:#24292f}Since the main feature of tiered storage is storing a
> large amount of data, we expect num_remote_segments to be large and a
> frequent linear scan could be expensive (depending on the underlying storage
> used by RemoteLogMetadataManager).
> Segment offloads and segment deletions are run together in the same task and
> a fixed size thread pool is shared among all topic-partitions. A slow logic
> for calculation of total_log_size could result in the loss of availability as
> demonstrated in the following scenario:{color}
> # {color:#24292f}Calculation of total_size is slow and the threads in the
> thread pool are busy with segment deletions{color}
> # Segment offloads are delayed (since they run together with deletions)
> # Local disk fills up, since local deletion requires the segment to be
> offloaded
> # If local disk is completely full, Kafka fails
> Details are in KIP -
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-852%3A+Optimize+calculation+of+size+for+log+in+remote+tier]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
