maytasm commented on PR #14642: URL: https://github.com/apache/druid/pull/14642#issuecomment-1648558611
- I believe KillUnusedSegments actually convert interval to segments before passing to SegmentNukeAction. (KillUnusedSegments uses RetrieveUnusedSegmentsAction to find unused segments for a given datasource + interval). So we actually started with an Interval. I am not sure if there is any reason that we are using segment ids rather than interval (as suggested by @kfaraz). - I am convinced we should make this change. KillTask can still be issued manually and the number of segments in each KillTask can easily be very large without the user realizing it. - For returning the segment actually deleted from SegmentNukeAction, I was thinking we can use this to determine which segments we want to delete from s3. (skip deleting from s3 if SegmentNukeAction failed to remove segment from metadatastore) - If any files fail to delete from S3, the segments are still gone from SQL -> I think this is less of a problem than if the file is deleted from s3 but still remains in SQL. If files fail to delete from S3, it does not effect Druid as Druid would considered them as deleted (gone forever). The s3 leak can be reconcile with s3 retention policy where the policy delete really old files (much greater than Druid retention policy to clean up the leak). if the file is deleted from s3 but still remains in SQL, Druid allow user to bring back unused segments (marking unused as used) which would cause problem if the actual files are no longer in deep stroage! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
