lifulong created SPARK-54000:
--------------------------------
Summary: Complex sql with expand operator and code gen enabled,
very slow
Key: SPARK-54000
URL: https://issues.apache.org/jira/browse/SPARK-54000
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 3.5.2
Environment: spark sql 3.5.2
Reporter: lifulong
Complex sql with expand operator and code gen enabled, very slow
sql format like select keya,keyb,count(distinct case when),...,count(distinct
case when),sum(a),sum(b) from x group by keya,keyb
when disable whole stage code gen, run will speed up 20x times
when add executor jvm parameter -XX:-TieredCompilation, run will speed up 20x
times
reduce select column count, such as 28 -> 27, can speed up 10x times
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
