gianm opened a new pull request, #18241: URL: https://github.com/apache/druid/pull/18241
Prior to this patch, queries from MSQ workers to realtime servers would be initiated in the processing pool, and would block the processing pool until results started coming in. This patch addresses it with this strategy: 1) Update DataServerClient to return a future that resolves when the response starts being written. 2) Split DataServerQueryHandler into DartDataServerQueryHandler and IndexerDataServerQueryHandler. The Dart version doesn't do retries and doesn't follow segments to other data servers. It just returns the async future from DataServerClient. The Indexer (tasks) version retains the prior logic and isn't really async. I didn't attempt to asyncify its retry logic in this patch. This is hopefully a situation that will be improved in the future, ideally by having realtime servers participate in MSQ queries as workers themselves. 3) Add ReturnOrAwait.awaitAllFutures, which allows processors to wait for a future to resolve. 4) Update ScanQueryFrameProcessor and GroupByPreShuffleFrameProcessor to give up the processing thread when waiting for a data server query to come back. Additionally, to simplify DataServerClient, cancellations are now issued without using a scheduled executor. There should be no need for this, because the service client is async. Tagged this as a bug because without this change, using Dart on Historicals would lead to the processing pool getting jammed up if there is realtime data in play. It seems serious enough to be classified as a bug. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
