Guangyuan Feng created KYLIN-5571: ------------------------------------- Summary: It takes too much time to calculate the data size during pushing down queries, which will lead to the queries un-stoppable. Key: KYLIN-5571 URL: https://issues.apache.org/jira/browse/KYLIN-5571 Project: Kylin Issue Type: Improvement Components: Query Engine Affects Versions: 5.0-alpha Reporter: Guangyuan Feng Assignee: Guangyuan Feng Fix For: 5.0-alpha
During pushing down the query, KE will try to calculate the included data size to set Spark partitions, but if there were too many files on HDFS, it will take a lot of time to complete. So in order to improve this situation, the following things will be done: # Using a limited thread pool to calculate the data size # Add timeout for the calculation, so as to stop the query as soon as possible After these changes, we can expected the query complete in a fixed duration. -- This message was sent by Atlassian Jira (v8.20.10#820010)