xiangfu0 commented on code in PR #19770:
URL: https://github.com/apache/pinot/pull/19770#discussion_r4225656994
##########
pinot-spi/src/main/java/org/apache/pinot/spi/config/table/SegmentsValidationAndRetentionConfig.java:
##########
@@ -88,6 +89,56 @@ public void setRetentionTimeValue(String retentionTimeValue)
{
_retentionTimeValue = retentionTimeValue;
}
+ /// Returns the size retention limit, expressed as a positive size
understood by
+ /// [DataSizeUtils#toBytes(String)], for example `"100G"`.
+ ///
+ /// The limit sums compressed archive bytes for active, live completed
segments, counted once regardless of
+ /// replication. Live-segment lineage filtering excludes sources of
COMPLETED entries and destinations of other
+ /// entries to avoid counting shadow copies. OFFLINE APPEND and REALTIME
limits are enforced independently, including
+ /// hybrid tables. Each typed table expires its own oldest eligible data
without consulting the OFFLINE routing
+ /// time boundary. REALTIME size retention can therefore expire data newer
than that boundary; hybrid time
Review Comment:
This sentence is the load-bearing one for hybrid operators, and it's
accurate — worth spelling out the query-visible consequence while you're here:
when size retention evicts a REALTIME segment newer than the routing boundary,
queries in that time range return no rows from either side (the data was never
moved to OFFLINE and is now gone from REALTIME), with no error. That's inherent
to any size cap, but for hybrid tables it's the one configuration where "size
cap" quietly means "data loss ahead of the migration task" — a sentence like
"for hybrid tables, size the cap so the REALTIME table comfortably holds all
data newer than the boundary, or RealtimeToOffline may lose its input" would
turn this from a semantics note into actionable sizing guidance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
