gianm opened a new issue, #20494: URL: https://github.com/apache/druid/issues/20494
The Broker runs a query itself, through LocalQuerySegmentWalker, when all of its datasources are global (e.g. inline, lookup, subquery, etc). On this path the Broker uses the same per-segment engines as data servers, but running on the query thread (the HTTP request handling thread) rather than a bounded processing pool. This can lead to excessive allocation of processing buffers, to the degree that the query engines request them. As a workaround, people can lower `druid.processing.buffer.sizeBytes` or raise `-XX:MaxDirectMemorySize` on Brokers. If doing the first, it may be useful to also enable disk spilling with `druid.query.groupBy.maxOnDiskStorage`. Possible fixes: 1. Give the Broker a bounded-size local processing pool, and run processing tasks there rather than in HTTP threads. 2. Allocate processing buffers by slicing up merge buffers, rather than allocating from an on-demand processing buffer pool. (Queries would wait for buffers to become available.) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
