Hi Chia-Ping, Anatolii, Kamal: Thanks everyone for the discussion and the helpful feedback. The proposed new duration configuration keeps the original functionality while adding more flexibility, which aligns well with what many of you suggested. So I adopt this approach. I’ve updated the KIP and the draft PR accordingly — please take another look when you have time and let me know if anything still needs adjustment. The main change is the revise of the configuration. Thanks again for the discussion!
Hi Luke and all: Based on the new feedback, I’ve updated the configuration part in the KIP. The previous configuration(boolean) has been revised to follows: remote.copy.lag.ms/log.remote.copy.lag.ms: 0: default value, no delay; -1: max delay (local.retention.ms); other values: (0, local.retention.ms] remote.copy.lag.bytes/log.remote.copy.lag.bytes: 0: default value, no delay; -1: max delay (local.retention.bytes); other values: (0, local.retention.bytes] Thanks Regards Jian Kamal Chandraprakash <[email protected]> 于2026年2月5日周四 11:29写道： > Hi all, > > The suggested config remote.log.copy.lag.ms fits nicely in the current > code > path too. It decouples the segment upload and local log deletion activity. > The copy lag ms upper bound should be upto local.retention.ms not more > than > it. > > On Thu, Feb 5, 2026, 02:39 Anatolii Popov via dev <[email protected]> > wrote: > > > Hi Jian, Chia-Ping, > > > > +1 on replacing the boolean with a duration config. > > > > Would it make sense to also add a bytes-based counterpart, e.g. > > remote.log.copy.lag.bytes? This would mirror how retention.ms and > > retention.bytes work together. > > > > The rationale: for topics with variable throughput, a time-based delay > can > > result in unpredictable upload backlogs. A bytes threshold gives > operators > > more direct control over maximum redundant storage per partition, which > > maps more directly to cost. > > > > Upload would trigger when either threshold is breached (whichever first), > > bounded by local retention in both dimensions. > > > > Best, > > Anatolii > > > > Anatolii Popov > > Senior Software Developer, *Aiven OY* > > m: +358505126242 > > w: aiven.io e: [email protected] > > <https://www.facebook.com/aivencloud> > > <https://www.linkedin.com/company/aiven/> < > https://twitter.com/aiven_io> > > > > On Wed, Feb 4, 2026, 9:04 PM Chia-Ping Tsai <[email protected]> wrote: > > > > > hi Jian > > > > > > Thanks for the proposal. I really like the cost-saving aspect, but I > feel > > > the current boolean flag might be too rigid for production environments > > > with large local retention. > > > > > > Would you consider replacing the flag with a duration config, say ` > > > remote.log.copy.lag.ms`? > > > > > > This would allow users to define a specific delay period (must be > between > > > 0 and `local.retention.ms`)? > > > > > > 1) If set to zero: current behaviour (upload immediately) > > > 2) if set to `local.retention.ms`: analogous to your proposed flag > > > (upload at the last moment) > > > 3) if set to a value in between: it creates a buffer windows. users can > > > save costs for a period but still upload data well before it hits the > > > retention limit, avoiding burst traffic issues > > > > > > This seems more flexible and safer for various use cases. > > > > > > Best, > > > Chia-Ping > > > > > > On 2025/11/19 13:29:11 jian fu wrote: > > > > Hi everyone, I'd like to start a discussion on KIP-1241, the goal is > to > > > > reduce the remote storage. KIP: > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1241%3A+Reduce+tiered+storage+redundancy+with+delayed+upload > > > > > > > > The Draft PR: https://github.com/apache/kafka/pull/20913 > Problem: > > > > Currently, > > > > Kafka's tiered storage implementation uploads all non-active local > log > > > > segments to remote storage immediately, even when they are still > within > > > the > > > > local retention period. > > > > This results in redundant storage of the same data in both local and > > > remote > > > > tiers. > > > > > > > > When there is no requirement for real-time analytics or immediate > > > > consumption based on remote storage. It has the following drawbacks: > > > > > > > > 1. Wastes storage capacity and costs: The same data is stored twice > > > during > > > > the local retention window > > > > 2. Provides no immediate benefit: During the local retention period, > > > reads > > > > prioritize local data, making the remote copy unnecessary > > > > > > > > > > > > So. this KIP is to reduce tiered storage redundancy with delayed > > upload. > > > > You can check the test result example here directly: > > > > https://github.com/apache/kafka/pull/20913#issuecomment-3547156286 > > > > Looking forward to your feedback! Best regards, Jian > > > > > > > > > >
