capistrant commented on PR #18844: URL: https://github.com/apache/druid/pull/18844#issuecomment-3755098774
> > For 2, I ended up going with the idea of a state being pending when it it inserted in to the database by the job queue. The overlord cleanup duty uses a secondary cleanup mechanism for pending compaction states, allowing operators to be more aggressive about cleaning up the unused states while not risking a compaction state being nuked before a long running compaction task completes. When inserting new segments, the coordinator will flip any associated compaction states from pending to "active" (pending=false) if needed. > > Yes, it makes sense to track the pending state in a separate column. I was about to suggest the same. 🙂 > > > The overlord cleanup duty uses a secondary cleanup mechanism for pending compaction states, > > I should think that the cleanup duty would be only a fallback mechanism for cleaning up pending compaction states. When a compaction job finishes (either SUCCESS or FAILED), the pending state associated with it should also be cleaned up. Hm, since a compaction state is associated with a fingerprint and not an individual task, I think having the task that fails be able to delete it is potentially dangerous. Assuming most datasources under compaction have multiple compaction candidates, multiple tasks could be associated with the same state and one erroneous failure that does a delete could lead to missing state for other compacted segments -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
