gortiz commented on code in PR #14946:
URL: https://github.com/apache/pinot/pull/14946#discussion_r1936947130
##########
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/MultiStageBrokerRequestHandler.java:
##########
@@ -112,6 +118,7 @@ public MultiStageBrokerRequestHandler(PinotConfiguration
config, String brokerId
CommonConstants.MultiStageQueryRunner.KEY_OF_MULTISTAGE_EXPLAIN_INCLUDE_SEGMENT_PLAN,
CommonConstants.MultiStageQueryRunner.DEFAULT_OF_MULTISTAGE_EXPLAIN_INCLUDE_SEGMENT_PLAN);
_queryThrottler = queryThrottler;
+ _queryCompileExecutor = Executors.newCachedThreadPool(new
NamedThreadFactory("query-compile-executor"));
Review Comment:
My reason for using a fixed number of threads is that optimizing a query is
a purely CPU-bound task. What is the advantage of having more threads than CPUs
in this case? In fact, I would use fewer threads than CPUs.
Imagine a situation where users are running tons of queries that are
expensive to parse. In that case, having a large number of threads running
optimizations would be expensive and make the system unresponsive. In the same
case, using a fixed thread pool smaller than the number of CPUs would queue up
queries, but other parts of the system shouldn't be affected.
Remember that Jersey can easily spawn hundreds of threads because it assumes
most work will be blocking. And IIRC, the new MSE query throttling mechanism is
applied after the query is optimized.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
