Hi all, The suggested config remote.log.copy.lag.ms fits nicely in the current code path too. It decouples the segment upload and local log deletion activity. The copy lag ms upper bound should be upto local.retention.ms not more than it.
On Thu, Feb 5, 2026, 02:39 Anatolii Popov via dev <[email protected]> wrote: > Hi Jian, Chia-Ping, > > +1 on replacing the boolean with a duration config. > > Would it make sense to also add a bytes-based counterpart, e.g. > remote.log.copy.lag.bytes? This would mirror how retention.ms and > retention.bytes work together. > > The rationale: for topics with variable throughput, a time-based delay can > result in unpredictable upload backlogs. A bytes threshold gives operators > more direct control over maximum redundant storage per partition, which > maps more directly to cost. > > Upload would trigger when either threshold is breached (whichever first), > bounded by local retention in both dimensions. > > Best, > Anatolii > > Anatolii Popov > Senior Software Developer, *Aiven OY* > m: +358505126242 > w: aiven.io e: [email protected] > <https://www.facebook.com/aivencloud> > <https://www.linkedin.com/company/aiven/> <https://twitter.com/aiven_io> > > On Wed, Feb 4, 2026, 9:04 PM Chia-Ping Tsai <[email protected]> wrote: > > > hi Jian > > > > Thanks for the proposal. I really like the cost-saving aspect, but I feel > > the current boolean flag might be too rigid for production environments > > with large local retention. > > > > Would you consider replacing the flag with a duration config, say ` > > remote.log.copy.lag.ms`? > > > > This would allow users to define a specific delay period (must be between > > 0 and `local.retention.ms`)? > > > > 1) If set to zero: current behaviour (upload immediately) > > 2) if set to `local.retention.ms`: analogous to your proposed flag > > (upload at the last moment) > > 3) if set to a value in between: it creates a buffer windows. users can > > save costs for a period but still upload data well before it hits the > > retention limit, avoiding burst traffic issues > > > > This seems more flexible and safer for various use cases. > > > > Best, > > Chia-Ping > > > > On 2025/11/19 13:29:11 jian fu wrote: > > > Hi everyone, I'd like to start a discussion on KIP-1241, the goal is to > > > reduce the remote storage. KIP: > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1241%3A+Reduce+tiered+storage+redundancy+with+delayed+upload > > > > > > The Draft PR: https://github.com/apache/kafka/pull/20913 Problem: > > > Currently, > > > Kafka's tiered storage implementation uploads all non-active local log > > > segments to remote storage immediately, even when they are still within > > the > > > local retention period. > > > This results in redundant storage of the same data in both local and > > remote > > > tiers. > > > > > > When there is no requirement for real-time analytics or immediate > > > consumption based on remote storage. It has the following drawbacks: > > > > > > 1. Wastes storage capacity and costs: The same data is stored twice > > during > > > the local retention window > > > 2. Provides no immediate benefit: During the local retention period, > > reads > > > prioritize local data, making the remote copy unnecessary > > > > > > > > > So. this KIP is to reduce tiered storage redundancy with delayed > upload. > > > You can check the test result example here directly: > > > https://github.com/apache/kafka/pull/20913#issuecomment-3547156286 > > > Looking forward to your feedback! Best regards, Jian > > > > > >
