xiangfu0 opened a new pull request, #19770: URL: https://github.com/apache/pinot/pull/19770
Tables currently retain segments only by time, so a burst can exceed a storage target while all segments remain within the TTL. Add opt-in `segmentsConfig.retentionSize` (for example, `"100G"`) to periodically evict the oldest eligible completed segments. The limit counts compressed segment archive bytes once, independently for each OFFLINE/REALTIME table, and can be used alone or alongside time retention. Preserve consuming/committing segments, the last completed LLC segments, and lineage-protected data. Skip size eviction when active metadata or sizes are incomplete; run after time and lineage cleanup to avoid excess deletion. Existing storage quotas retain their upload-admission behavior. A protected size-resolution hook supports subclasses with logical segment groups. Validation: all 125 targeted tests passed (`TableConfigBuilderTest`: 5; `TableConfigUtilsTest`: 84; `RetentionManagerTest`: 36), with no failures or skips. Scoped Spotless, Checkstyle, and license checks passed. Local test builds used a temporary compiler classpath addition for the existing missing JetBrains annotation dependency in `ZstandardDecompressor`; no build dependency changes are included. Release note: add optional `segmentsConfig.retentionSize` for periodic, oldest-first table retention by compressed segment size. OFFLINE APPEND tables and completed REALTIME segments are supported; protected data can leave the table above the configured limit. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
