yuan created KYLIN-5401:
---------------------------
Summary: Optimize code logic for pushdown queries
Key: KYLIN-5401
URL: https://issues.apache.org/jira/browse/KYLIN-5401
Project: Kylin
Issue Type: Improvement
Components: Query Engine, Spark Engine
Affects Versions: v4.0.3, v4.0.2, v4.0.1
Reporter: yuan
It is found that the push-down query in kylin4.0.x is very slow for the simple
query eg: {{{}select * from table limit 10{}}}. It should respond in seconds,
but it often takes a few minutes, and the larger the query data set, the more
time-consuming it takes Long, which is very abnormal.BI tools often execute
some simple queries to display detailed data. Abnormal query duration often
causes BI tools to time out and return error messages, which is very unfriendly
to user experience.
Through investigation, it is found that there is a *shuffle* process in the
query plan of this very simple detailed query, which is outrageous.
The main logic of Kylin executing push-down query is concentrated in
{{{}org.apache.kylin.query.pushdown.SparkSqlClient{}}}.Unnecessary Spark
DataFrame type transform in
{{org.apache.kylin.query.pushdown.SparkSqlClient#DFToList}} is the main cause
of this problem.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
