Nagarjunanm commented on issue #20494: URL: https://github.com/apache/druid/issues/20494#issuecomment-6268062562
Thanks for the feedback and yes having lesser configuration make sense which is in inline with how Druid has evolved over years even in ingestion front as well. Here is the rough thought on the same [ option 2 ]: - When the Broker runs a query itself, it would take the memory from its merge buffers instead of keeping a separate processing buffer pool. Its memory check would then only count merge buffers, so no Broker would need more memory than it does today. - Groupby is the main query type that needs a processing buffer here. The catch is that it's already holding a merge buffer while it runs. If it then had to wait for a second merge buffer, queries could end up stuck waiting on each other. So instead, it would split the merge buffer it already holds, using part of it for processing and the rest for merging. - Timeseries only needs a buffer when it vectorises, so it would borrow a merge buffer in that case and wait if none is free. That's safe because it isn't holding any other buffer while it waits. topN doesn't need one I believe. Follow up: Is it fine with groupBy splitting the merge buffer it already holds? The other option is for each local groupby to take an extra merge buffer at the same time as its first one, when the query starts. That avoids the waiting problem and keeps buffers full-size, but each local groupBy would need one more merge buffer than it does now. Please do let me know if I'm missing anything here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
