hi Jian Thanks for the proposal. I really like the cost-saving aspect, but I feel the current boolean flag might be too rigid for production environments with large local retention.
Would you consider replacing the flag with a duration config, say `remote.log.copy.lag.ms`? This would allow users to define a specific delay period (must be between 0 and `local.retention.ms`)? 1) If set to zero: current behaviour (upload immediately) 2) if set to `local.retention.ms`: analogous to your proposed flag (upload at the last moment) 3) if set to a value in between: it creates a buffer windows. users can save costs for a period but still upload data well before it hits the retention limit, avoiding burst traffic issues This seems more flexible and safer for various use cases. Best, Chia-Ping On 2025/11/19 13:29:11 jian fu wrote: > Hi everyone, I'd like to start a discussion on KIP-1241, the goal is to > reduce the remote storage. KIP: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1241%3A+Reduce+tiered+storage+redundancy+with+delayed+upload > > The Draft PR: https://github.com/apache/kafka/pull/20913 Problem: > Currently, > Kafka's tiered storage implementation uploads all non-active local log > segments to remote storage immediately, even when they are still within the > local retention period. > This results in redundant storage of the same data in both local and remote > tiers. > > When there is no requirement for real-time analytics or immediate > consumption based on remote storage. It has the following drawbacks: > > 1. Wastes storage capacity and costs: The same data is stored twice during > the local retention window > 2. Provides no immediate benefit: During the local retention period, reads > prioritize local data, making the remote copy unnecessary > > > So. this KIP is to reduce tiered storage redundancy with delayed upload. > You can check the test result example here directly: > https://github.com/apache/kafka/pull/20913#issuecomment-3547156286 > Looking forward to your feedback! Best regards, Jian >
