[
https://issues.apache.org/jira/browse/KYLIN-6033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17928313#comment-17928313
]
Guoliang Sun commented on KYLIN-6033:
-------------------------------------
h3. Current Status
Kylin currently only supports constant queries using the Calcite engine, while
all other queries are handled by the Sparder engine. See:
`org.apache.kylin.query.engine.QueryExec#executeQueryPlan`.
For non-constant queries, Kylin first divides the query into OLAPContexts and
then generates a Spark query plan.
When generating the Spark query plan, Kylin checks whether the subquery
corresponding to the current OlapContext can be answered using metadata. See:
`org.apache.kylin.query.runtime.CalciteToSparkPlaner#convertTableScan` and
`org.apache.kylin.query.runtime.CalciteToSparkPlaner#convertJoinRel`.
h3. Solution (Use Metadata to Answer Queries with Calcite Engine)
For non-constant queries, after dividing the OLAPContext but before generating
the Spark query plan, insert logic to check if the entire query can be answered
solely using metadata. If so, switch to the Calcite engine for execution.
h3. Existing Limitations
Since a single query can only use one query engine for execution, the Calcite
engine can only be used when all OLAPContexts split from the query can be
answered using metadata.
When all OLAPContexts of a query can be answered using metadata, executing with
the Calcite engine has the following issues:
1. The `min` and `max` values of dimensions in the metadata are stored as
strings and cannot be directly used as input for the Calcite engine. They need
to be converted to the actual table column types. Otherwise, queries with `min`
and `max` functions on non-string type parameters will fail.
2. When multiple subqueries are combined via `UNION`, and the `UNION` itself is
treated as a subquery, Calcite requires the field types of the `UNION`
subqueries to be consistent. Kylin3 aligns subquery field types using
`UnionTypeCastRule`, but Kylin4 lacks this rule.
The above issues will be fixed in this issue.
> When answering `min` and `max` queries using metadata, route to the Calcite
> engine to avoid submitting Spark tasks
> ------------------------------------------------------------------------------------------------------------------
>
> Key: KYLIN-6033
> URL: https://issues.apache.org/jira/browse/KYLIN-6033
> Project: Kylin
> Issue Type: Improvement
> Affects Versions: 5.0.0
> Reporter: Guoliang Sun
> Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
