kfaraz commented on PR #18687: URL: https://github.com/apache/druid/pull/18687#issuecomment-3449377518
I guess it is natural to allow more start up strategies since we already support - loadAllLazily and loadAllEagerly. But the approach here feels a little complicated. We are also trying to re-invent the things which load rules already do. It would perhaps make more sense to use "Virtual Storage Fabric" + load rule combo since they are much more powerful and will only evolve over time. IIUC, the use case here is that we frequently query the data of the last `x` days and also query the data of the last `x + y` days but less frequently so. Rather than doing the eager vs lazy load of the segments in the `y` days only on startup, can we _always_ load these segments lazily? So, maybe the solution could be a mix of the recently added "Virtual Storage Fabric" + some load rules. Load rules would perhaps look something like: Rule 1: `loadByPeriod` last 7 days (i.e. load eagaerly on startup and otherwise) Rule 2: `loadLazilyByPeriod` last 30 days (i.e. these segments are present on virtual storage fabric and will be loaded when the first query is received for them. Coordinator remains agnostic of this fact. The virtual storage can also be reclaimed to load other virtual segments.). Rule 3: `dropForever` any data beyond the last 30 days. @FrankChen021 , @GWphua , would this satisfy your use case? @clintropolis , @abhishekrb19 , what are your thoughts? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
