swaminathanmanish commented on code in PR #18518:
URL: https://github.com/apache/pinot/pull/18518#discussion_r3265697379
##########
pinot-common/src/main/java/org/apache/pinot/common/lineage/SegmentLineageUtils.java:
##########
@@ -53,4 +54,35 @@ public static void
filterSegmentsBasedOnLineageInPlace(Set<String> segments, Seg
}
}
}
+
+ /**
+ * Returns the set of segments that participate in a live lineage entry and
therefore must not be deleted by
+ * external callers.
+ * <ul>
+ * <li>{@code IN_PROGRESS} entries lock both {@code segmentsFrom} and
{@code segmentsTo}.</li>
+ * <li>{@code COMPLETED} entries lock {@code segmentsFrom}</li>
+ * <li>{@code REVERTED} entries lock nothing.</li>
+ * </ul>
+ */
+ public static Set<String> getDeleteBlockedSegments(SegmentLineage
segmentLineage) {
+ if (segmentLineage == null) {
+ return Set.of();
+ }
+ Set<String> blocked = new HashSet<>();
+ for (LineageEntry lineageEntry :
segmentLineage.getLineageEntries().values()) {
+ switch (lineageEntry.getState()) {
+ case IN_PROGRESS:
+ blocked.addAll(lineageEntry.getSegmentsFrom());
+ blocked.addAll(lineageEntry.getSegmentsTo());
+ break;
+ case COMPLETED:
Review Comment:
> If Seg1 is replaced by Seg2 via lineage, and If we don't include COMPLETED
here, then the delete API returns success and thus the client assumes the
delete is succesful but in reality it is not (the deleted data is served by
Seg2)
>
> I can make changes such that forceDelete can delete segments in FROM list
of COMPLETED lineage. But that requires many changes because delete API doesn't
take any such parameter today.
>
> Adhoc, these segments can be deleted by triggering RetentionManager.
>
> But I think it makes sense for default behavioir to fail if its part of
FROM list of COMPLETED lineage.
>
> Let me know your thoughts
It makes sense to have deterministic behavior. Triggering retention manager
is the only way to clean up lineages.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
