shuturmurgh commented on code in PR #18486:
URL: https://github.com/apache/pinot/pull/18486#discussion_r3240735448
##########
pinot-common/src/main/java/org/apache/pinot/common/utils/RetentionUtils.java:
##########
@@ -54,18 +61,40 @@ private RetentionUtils() {
*/
public static boolean isPurgeable(String tableNameWithType,
SegmentZKMetadata segmentZKMetadata, long retentionMs,
long currentTimeMs, boolean useCreationTimeFallback) {
- String segmentName = segmentZKMetadata.getSegmentName();
- long endTimeMs = segmentZKMetadata.getEndTimeMs();
+ return isPurgeableInternal(segmentZKMetadata.getEndTimeMs(),
segmentZKMetadata.getCreationTime(), retentionMs,
+ currentTimeMs, useCreationTimeFallback, tableNameWithType,
segmentZKMetadata.getSegmentName());
+ }
+
+ /**
+ * Whether segment file metadata is past the retention window, using end
time from
+ * {@link #segmentMetadataEndTimeMillis(SegmentMetadata)} and {@link
SegmentMetadata#getIndexCreationTime()} as the
+ * optional fallback clock. Invalid-end paths emit the same style of
debug/warn lines as the ZK overload, using
+ * {@link SegmentMetadata#getName()} for the segment in log messages.
+ * <ul>
+ * <li>If end time is valid: purgeable when {@code currentTimeMs -
endTimeMs > retentionMs}.</li>
+ * <li>If end time is invalid and {@code useCreationTimeFallback} is true
and index creation time is valid:
+ * purgeable when {@code currentTimeMs - creationTimeMs >
retentionMs}.</li>
+ * <li>Otherwise: not purgeable (fail-open).</li>
+ * </ul>
+ */
+ public static boolean isPurgeable(String tableNameWithType, SegmentMetadata
segmentMetadata, long retentionMs,
+ long currentTimeMs, boolean useCreationTimeFallback) {
+ return isPurgeableInternal(segmentMetadataEndTimeMillis(segmentMetadata),
segmentMetadata.getIndexCreationTime(),
+ retentionMs, currentTimeMs, useCreationTimeFallback,
tableNameWithType, segmentMetadata.getName());
+ }
+
+ private static boolean isPurgeableInternal(long endTimeMs, long
creationTimeMs, long retentionMs,
+ long currentTimeMs, boolean useCreationTimeFallback, String
tableNameWithType,
+ String segmentName) {
if (TimeUtils.timeValueInValidRange(endTimeMs)) {
return currentTimeMs - endTimeMs > retentionMs;
}
+ if (useCreationTimeFallback &&
TimeUtils.timeValueInValidRange(creationTimeMs)) {
Review Comment:
For single-segment upload we now call `RetentionUtils.isPurgeable(...,
useCreationTimeFallback=false)`, so we only use the segment’s data end time
from file metadata.
Invalid/missing end time remains fail-open (no reject).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
