kfaraz commented on PR #18844: URL: https://github.com/apache/druid/pull/18844#issuecomment-3753749750
> For 2, I ended up going with the idea of a state being pending when it it inserted in to the database by the job queue. The overlord cleanup duty uses a secondary cleanup mechanism for pending compaction states, allowing operators to be more aggressive about cleaning up the unused states while not risking a compaction state being nuked before a long running compaction task completes. When inserting new segments, the coordinator will flip any associated compaction states from pending to "active" (pending=false) if needed. Yes, it makes sense to track the pending state in a separate column. I was about to suggest the same. 🙂 > The overlord cleanup duty uses a secondary cleanup mechanism for pending compaction states, I should think that the cleanup duty would be only a fallback mechanism for cleaning up pending compaction states. When a compaction job finishes (either SUCCESS or FAILED), the pending state associated with it should also be cleaned up. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
