maytasm commented on pull request #11025: URL: https://github.com/apache/druid/pull/11025#issuecomment-812212391
> > In a Coordinator run, the RunRules happen before the UnloadUnusedSegments. Hence, we should be loading the new segments before dropping the existing segments. > > I'm not sure that it's guaranteed that this will happen. Some scenarios that come to mind: > > * It's possible the historical load queues are maxed out (they are at maxSegmentsInNodeLoadingQueue) and so the coordinator may not issue load commands for all of the new segments. > * I think segment loading is asynchronous, so even if all load commands are issued, they might not have finished executing by the time UnloadUnusedSegments runs. You're right. Seems like there is no easy way of marking old segments as used=false before the new segments are loaded. Changing logic in loading/dropping will be too complicated. Maybe we should make it generate empty segments with new granularity for all the missing gaps. And let the loading/dropping handle it as normal. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
