kfaraz commented on PR #18687: URL: https://github.com/apache/druid/pull/18687#issuecomment-3451062046
Thanks for the clarification, @FrankChen021 ! I agree that the solution here is different from what vsf + load rule would do. But I was wondering if it was close enough to satisfy your use case. That is, why not delay downloading the segment from the deep storage too? > the implementation here is not re-invent, and not complicated. > Making a small change to existing segment cache loading is still worthy. True, as mentioned, I am not opposed to the idea of new startup cache load strategies. It only seems natural. But the additional configs mean more maintenance work for cluster admins. They would need to think about which segments they want to be loaded on the cache eagerly and which are okay to be left for later. It is a question (kind of) similar to load rules i.e. which segments to keep on historicals and which are okay to be left just on the deep store. The page cache is just the first level of caching, the second being the disk of the historical itself. Since load rules already work well with the concept of period-based loading, I hoped it would be more useful for the future to just extend that concept to cover such use cases as well. But if that doesn't cover your use case, I can understand. > If the goal of virtual storage will be the dominant mode in future, introducing loadLazilyByPeriod to loading rules makes sense. > Virtual storage is just merged and is still under experiemental, I don't know when it will be production ready, and what's the roadmap for it. I think @clintropolis would have some insights there but I imagine it should see good adoption in the near future. I was hoping you would be one of the early birds! 😉 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
