[ https://issues.apache.org/jira/browse/KYLIN-5571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Guangyuan Feng updated KYLIN-5571: ---------------------------------- Fix Version/s: 5.0-beta (was: 5.0-alpha) > It takes too much time to calculate the data size during pushing down > queries, which will lead to the queries un-stoppable. > ---------------------------------------------------------------------------------------------------------------------------- > > Key: KYLIN-5571 > URL: https://issues.apache.org/jira/browse/KYLIN-5571 > Project: Kylin > Issue Type: Improvement > Components: Query Engine > Affects Versions: 5.0-alpha > Reporter: Guangyuan Feng > Assignee: Guangyuan Feng > Priority: Major > Fix For: 5.0-beta > > > During pushing down the query, KE will try to calculate the included data > size to set Spark partitions, but if there were too many files on HDFS, it > will take a lot of time to complete. > So in order to improve this situation, the following things will be done: > # Using a limited thread pool to calculate the data size > # Add timeout for the calculation, so as to stop the query as soon as > possible > # Add new properties: > {color:#4c9aff}_kylin.query.pushdown.auto-set-shuffle-partitions-multiple=3_{color},the > default Spark partition num > _{color:#4c9aff}kylin.query.pushdown.auto-set-shuffle-partitions-timeout=30{color},_ > the maximum timeout, 30 seconds by default, to calculate the data size in > order to adjust the Spark partition num > After these changes, we can expected the query complete in a fixed duration. -- This message was sent by Atlassian Jira (v8.20.10#820010)